SMF2 to Flarum migration script - alpha 3

marcozambi · 2016-08-18T13:39:21+00:00

smf2_to_flarum Migration script for SMF2 forum to Flarum, partially based on the phpbb_to_flarum script by robrotheram, VIRUXE, Reflic https://github.com/robrotheram/phpbb_to_flarum Where to download it You find the script on GitHub https://github.com/ItalianSpaceAstronauticsAssociation/smf2_to_flarum Description The Script exports and migrates your SMF2 (possibly SMF1.x) forum to Flarum. It supports: - a DB to DB migration, - a DB to ASCII SQL file export. These two functionalities can be performed together or as alternatives. Flarum is still in beta testing, therefore only some of the typical web forum features are available, and what is now working can break anytime (including this script, tailored for flarum ver. 0.1.0-beta.5). At this moment smf2_to_flarum only supports migration of: - Users and their avatars (no passwords are exported. Migrated users will have to ask for a new one). - Boards and sub boards (Categories are intentionally ignored, and boards deeper than 2nd level are added in Flarum as extra Tags) - Topics and messages (with some bbcode to flarum-markdown translation) - Attachments (partial, only images) Changelog 2016-08-28: Version 0.3 Alpha Added preliminary image attachments export General bugfix. 2016-08-27: Version 0.2 Alpha Added preliminary avatars export Added alternative script version, based on s9e\TextFormatter library (http://s9etextformatter.readthedocs.io). General bugfix. 2016-08-26: Version 0.1 Alpha First commit (for flarum ver. 0.1.0-beta.5) Known issues / Future features Only SMF/SMF2 image attachments are exported. While SMF supports all kind of attachments, Flarum only allows image attachments and provided that flagrow-image-upload extension has been installed (see #1836) Private Messages are not exported yet, since this feature is not yet supported by Flarum. Smilies are not migrated. Installation Install a vanilla Flarum, with empty database. Copy smf2_to_flarum.php (or smf2_to_flarum_alternative.php) and smf2_to_flarum_settings.php in your SMF forum root. Customise smf2_to_flarum_settings.php script settings with the correct username/password for your SMF2 and (optionally) Flarum DB, and enable/disable the export settings (all enabled by default). Run the script as http://www.yoursmf2website.xyz/smf2_to_flarum.php Alternative version The alternative version of the script (smf2_to_flarum_alternative.php) needs the following tools to be installed on the same server where SMF2 is installed: - Composer: https://getcomposer.org/download/ - s9etextFormatter: http://s9etextformatter.readthedocs.io/Getting_started/Installation/ (should be in the script's dir) Rever to the respective websites for install instructions. Avatars The script saves users' avatars in a directory named "avatars". From there the avatars shall be manually copied into flarum "assets/avatars" directory. Attachments Attachment management is not part of flarum core. In order to have attachments functionality in flarum the "Image Upload" extension has to be installed and enabled (see #1836). The migration script exports only image attachments, since no other attachment type is supported at the moment. Files are extracted from single or multiple directory attachment system of SMF/SMF2 then stored in a directory named "images". From this directory the attachments shall be manually copied in "assets/images" directory of your flarum installation. Help us Feel free to contribute to this project! About us Hello, my name is Marco and I'm part of the technical board of forumastronautico.it, the largest web forum in Italian dedicated to space enthusiasts. Our forum has over 1300 users and 270000 messages. We are quite happy with SMF itself, but as many long standing web forum platforms, it's starting to fall behind in terms of feature, responsiveness, web2.0-compliance. We started to look for the "next thing" and flarum looks really really amazing. In order to test Flarum funcionalities in a proper way we decided to migrate our entire forum to Flarum, in a special test environment. This script is the result of the need to move our users, boards hierarchy and topics/messages to the new platform.

-ExotiC-

@marcozambi So you have any idea what the problem could be? I still can not find a flaw in the code ?

acseven

-ExotiC- @marcozambi
I'm also trying to use this script to test the migration of a SMF board to Flarum and came across this issue (and a bunch of others). Although I can't say why, the user list is being limited by the last part of the query (the WHERE part):

WHERE 
			".$table_prefix."attachments.attachment_type = 0

If you remove that, it will at least select every member instead of just a few (in my case ~100 in a little over 10000).

On other issues:

I don't think anyone has mentioned importing attachments other than images using flagrow/upload, the replacement for flagrow/flarum-ext-image-upload which the script is using. I have no idea on how to approach this, so if anyone has any pointers I'd be welcome to try.
This line makes no sense: $src_dir = $attachments_dir[$idx_src_dir];and I replaced it by $src_dir = $attachments_dir[0]; and along with removing the WHEREstatement I mentioned earlier makes the import output clean and tidy. But making that change leaves me thinking what is the $idx_src_dirvariable for.

That said, so far I've imported Users, Categories (boards/sub-boards) and Posts. I'm going to check on these to see if everything is passing on as it should or not, and later on I'll try the attachments part.

Edit: Posts imported more or less ok, but it timed-out and I'm yet to verify what's missing.

0E800

acseven Is the SMF bbcode, preserved in the posts? Links? Images?

acseven

0E800
Last Edit on: 24th Mar 2017

For organisation purposes, I'm editing this post as I go along using the Alternative version of the script and adding short new posts referring back here:

SCRIPT USAGE RESULTS:

Users:

All Users are imported to the same Member group (group id 3): i.e. you'll have no admins, mods, etc., you'll have to set those manually, and it will be overwritten every time you run the script (unusable password hash included)
Avatars: ?? None of my forum's avatars were imported (does the script process them or not? Also, there are local avatars and external ones!)
User Signatures in Posts: No (missing from flarum?)_

Boards:

Permissions: All of those will be gone, permissions default for all categories

Posts - Text (mostly bbcode):

Post migration: it works, but my database is quite large and the script breaks every time. To be checked later on
Paragraph Text: Mostly Ok, though there is something off with the paragraph spacing, it doesn't seem to be as neat as the one in these posts at this forum. Edit: the script is converting double line break tags   to single line  , making posts less readable (no spacing between paragraphs). Edit #2: it's by default in the script, read mode here
Quotes: Ok as well as Quotes within Quotes
Links: Mostly Ok, BUT the internal URLs are not being corrected. That's to be fixed with rewrites, I guess, but it would be great if the URLs could migrate to the new Flarum URL...
Code: Ok
Bold text: Ok
The script removes all non converted BBcode by default (// Delete all non converted bbcode)

Posts - Media/Other:

Polls: No, polls are not available in flarum, read more here
Embedded Youtube Video URLs: Ok
Images:
- I have yet to run this part of the script, as I would like to try using the flagrow/upload extension instead (to import all attachments, not only images)
- However, images inline to posts are displaying correctly, while linked to their original location/URL
Other Attachments: not possible with the current extension the script supports, needs to be updated to use flagrow/upload

AFTER MIGRATION BOARD FUNCTIONALITY:

Board Permissions: to check
User (Guest) Permissions: to check
Support for Google Analytics
...

INSTALLED EXTENSIONS:

Unrelated to the script, but as I am testing it I should mention I have installed these extensions so far, and they seem to work well:

Flagrow/Ads (github): working
Flagrow/Upload (github): working , though no drag and drop which seems to be a feature
xengine/flarum-ext-markdown-editor or SimpleMDE for Flarum: working, though it would be nice if it would integrate with flagrow/upload
Mediaembed
Flagrow/flarum-ext-image-upload (deprecated, installed just for the SMF-to-Flarum migration script)
Flagrow/bazaar

acseven

One additional thing: the script keeps timing-out when importing Posts. My database is pretty large, so that's the reason, but still there should be a way of knowing what's happening and limiting it.

acseven

I've checked what I mentioned earlier about the paragraphs being off: there is definitely something wrong, as (for example) double break tags (   ) are being imported as only one.

Edit: found it in the code, it's there by definition (but in my case i'll comment it out):
// Replace multiple (one ore more) line breaks with a single one. $text = preg_replace('/[\r\n]{2,}/s', "\n", $text);

Ralkage

The script will need some continuous work when Flarum releases a new build, in which case extensions are inclined to break because of naming convention changes, API changes, etc.

acseven

Ralkage
I wouldn't know how to comment on that with certainty, but I would say that most of what is getting migrated (my log here) is not directly dependant on extensions after migration (maybe the youtube videos?)... In any case, it works pretty well as is, so far, which may be enough for many users.

acseven

One additional information: the only way I apparently would be successful in importing Posts, without timeouts, will by disabling the SQL file dump variable $do_dump = false;. This might have to do with the fact that my server is using nginx and [apparently] if there is no browser output after some time (which happens during the file dump), it will eventually fail with a timeout. I wrote "would be" because the whole table is over 130k messages and the estimated time to complete the process using the script remotely is about 7 hours.
Whenever (if) I decide I can migrate I'll have to figure a better way to do the transfer; I'll probably do it on local DB copies but it would be useful if the script at least had an option to use an existing SQL dump file already on the server, instead of creating one every time.

EvelinaAdamovi

@acseven can you show for us how yours smf2_to_flarum.php file look like? With your changes?

acseven

EvelinaAdamovi

I'm using the alternative version of the script, and here's the changes I'm using - note that I have not tested importing attachments (i.e. images):

acseven Commented this out:

// Replace multiple (one ore more) line breaks with a single one. $text = preg_replace('/[\r\n]{2,}/s', "\n", $text);

acseven Removed this from the User query:

WHERE ".$table_prefix."attachments.attachment_type = 0

acseven

Replaced:

$src_dir = $attachments_dir[$idx_src_dir];

by this

$src_dir = $attachments_dir[0];

EvelinaAdamovi

Hmmm... I'm now thinking what about old urls in forum posts. I mean:

/index.php?topic=1161

now will be not accessible, And truly can't now imagine how solutions may look like for this.

acseven

EvelinaAdamovi
I think one viable way would be for the script to convert those as well in the process of migration, or at least afterwards. I thought of rewrites earlier but that won't work as the DB structure will be different. The migration process could create a temporary table (?) correlating a given message (and topic) in both the SMF and Flarum databases and afterwards be used to lookup the correct values to be replaced in URLs.

marcozambi

Hello everybody.
Just to say thanks for your interest in this migration script. Although I haven't written any updates for a long time, I want to clarify that I haven't abandoned at all the development and I'm just waiting for the first stable to be released to complete the work and merge some fixes I coded in the last few months.
Once the first stable is out, I'll get back to work ?
Thanks for sharing your experiences!

acseven

marcozambi
Thanks for your work and your comment! I had read another comment earlier where you mentioned waiting for a stable release, but in case it takes a lot of time and if you want something tested, let us know!

EvelinaAdamovi

I agree with @acseven, if you have something in mind, share it ?

Thanks for your effort.

acseven

marcozambi just a heads up on polls ? up: #53107

acseven

For future reference, someone else delved into this:
https://github.com/sriharshachilakapati/JGO-Flarum-Migration
This migration script appears to handle only boards, categories and users.
Edit: and for SMF 1.1

SHC

acseven It's interesting that I found my attempt here. I'm still working on it to include posts.

However, this script might not work for everyone, as my current backend is a heavily modified SMF, that too a very old one. It's 1.1.18 to be precise.

For SMF 2.x, this won't work as there are a lot of changes in those SMF versions.

acseven

SHC I found it a while ago and remembered to add it to the thread, as it might come handy when the first stable release of flarum is released. But thanks for the additional notes!

« Previous Page Next Page »