SMF2 to Flarum migration script - alpha 3

wolfman

Thanks for the script! ?

I run an old SMF1 forum which i want to migrate to Flarum - or at least like @marcozambi test Flarum with the original content from SMF1. This script seems to be a good starting point for an SMF1-migration script. I will look at this in a couple of days.

marcozambi

wolfman In the next few days we're going to release further updates. We are currently working on:
- Detecting and converting youtube links into embedded videos (this will work if you have Mediaembed extension installed)
- Smilies conversion to http://emoji.codes/ standard

Later we plan to fix the avatar export, and most probably the image attachments.

wolfman

wolfman I run an old SMF1 forum which i want to migrate to Flarum - or at least like @marcozambi test Flarum with the original content from SMF1. This script seems to be a good starting point for an SMF1-migration script. I will look at this in a couple of days.

Well, couple of days turned into almost 2000 days! But better late then never i guess! 😉 So i finally migrated my SMF1 forum this morning with about 25k topics/336k posts to Flarum and everything went fine so far!

Instead of using the script from @marcozambi as initially intended, i went with the script provided by SHC (see also https://discuss.flarum.org/d/7290-migration-for-a-very-large-smf-forum and https://github.com/sriharshachilakapati/JGO-Flarum-Migration) as the basis for my migration script as it better fitted my needs.

I added some additional functionality to the Script like migrating avatars or importing file-attachments to FoF Upload via Flarum-API, converting internal links, replacing Smileys with Emojis or migrating boards to secondary tags. While some adjustments might be of wider use, others are oddly specific for my migration!

However, if somebody is still running a SMF-Forum and wants to migrate to Flarum (highly recommended!), this might be a good starting point for your migration: https://github.com/wolfgang-demeter/smf2flarum
It will however not run out of the box for you!

I migrated a SMF1 Forum. I do not know if this will work or is compatible with SMF2!

meetdilip

What about polls ?

marcozambi

meetdilip well, the problem is that at the moment afaik flarum does not offer the polls functionality nor in the core neither as extension, so we have literaly no way to port it.
We'll have to wait for that, or start to develop an extension on our own, but we personally have no plan for that.

acseven

marcozambi just a heads up on polls ? up: https://discuss.flarum.org/d/5310-reflar-revolutionizing-flarum/7

JoshyPHP

marcozambi If we have to pick one as most important, I would go with [quote] bbcode and nested [quote]. Are they supported at all in flarum?

They are and they can be nested up to 10 levels deep by default. Flarum's quotes don't carry as much information as SMF's. They could be changed via an extension to resemble SMF's more, but I'm not sure that's worth the effort.

luceos why not rewrite to markdown?

It's simpler and less likely to introduce inconsistencies if you the original text as-is. Since Flarum supports BBCodes it should be easier too.

marcozambi Now, of course we try to look behind the curtains, and if I create a post and enter Text in bold as text, than post it, clearly this gets "translated" and saved as:

<r><p><STRONG><s>**</s>Text in bold<e>**</e></STRONG></p></r>

That's XML, generated by the text formatting library that handles BBCodes/Markdown/etc... I'm its author and I keep an eye on Flarum to see how it's used and anticipate users' needs. I would definitely advise against trying to generate the XML directly. If you make any mistake or if there's a special case you didn't foresee you risk making it invalid, and thus unusable.

Instead, grab the original text from SMF (the same text a user would get when editing the post) and use Flarum to parse it. I don't know how you're supposed to get a hold of Flarum's Formatter object from outside the application but I assume you can bootstrap a minimal app and call $this->app->make('flarum.formatter'). You'll have to ask the Flarum devs about it.

If you use Flarum's Formatter object then it will automatically handle YouTube links if a YouTube extension is installed. You may also want to create an extension that adds the missing BBCodes from SMF to Flarum. The library was designed to be able to parse BBCodes from basically any forum software with high fidelity, so it's up to you how close you want your new Flarum to be in terms of BBCodes and other markup compared to SMF. I'm subscribed to this thread so if you have any question about markup, feel free to post them here.

If there's enough demand for it, I may consider publishing a Flarum extension that adds the missing BBCodes and markup. That would require a bunch of research and development so it's unlikely at the moment.

marcozambi

JoshyPHP Thank you for your reply.
I managed to understand how to translate most of the SMF bbcode into flarum XML, and I can migrate correctly almost everythig with the script we released yesterday.
It makes a lot of sense to not re-invent the wheel though, so I'll definitively try to update it in the next few days to make use of your beautiful library.

marcozambi

The script has been updated with basic avatar export and an alternative version making use of JoshyPHP s9e\TextFormatter tool.

marcozambi

JoshyPHP Hi there! I'm working again on the code for a better Board-to-Tag conversion, and I decided to abandon the "do-it-yourself" bbcode to XML translation and go fully with your fantastic library s9e/TextFormatter.

I'm using the Forum bundle at the moment, with excellent results, but I noticed that for some reason the emoticons/emoji are not converted.
I tried an experiment, doing a small quick conversion using, instead of the Forum bundle, only the emoji Plugin.

If I try this code

$configurator = new s9e\TextFormatter\Configurator;
$configurator->Emoji->useEmojiOne();
extract($configurator->finalize());

they are converted perfectly.

I was wondering, is there a way (I tried all sort of things) to tweak some settings in the Forum bundle,to set explicitly for example the usage of Emoji->useEmojiOne(); in the Bundle itself?

You did a superb job with this library, but if I may suggest you an improvement that would immensely help potential users,, some extra usage about "mixing" Bundles with Configurator examples would be great!

Thanks in advance!

marcozambi

Added preliminary image attachments export, and some bugfixing.

JoshyPHP

The Emoji plugin was only added recently to the Forum bundle, it's possible your version doesn't include it. Bundles are read-only snapshots of a particular configuration. It's not possible to modify them.

However, you can get a configured instance of s9e\TextFormatter\Configurator that you can use to generate your own instance of the parser and render. You can also save them as your own bundle if you want to reuse them more easily. Or you may want to just serialize the parser/renderer and reuse those instead, as Flarum does. That's the preferred method.

$configurator = s9e\TextFormatter\Configurator\Bundles\Forum::getConfigurator();
$configurator->Emoji->addAlias('x.x', '😵');
$configurator->saveBundle('MyBundle', '/tmp/MyBundle.php');

include '/tmp/MyBundle.php';

$text = '[center]x.x[/center]';
$xml  = MyBundle::parse($text);
$html = MyBundle::render($xml);

die("$html\n");

<div style="text-align:center"><img alt="x.x" class="emoji" draggable="false" width="16" height="16" src="//cdn.jsdelivr.net/emojione/assets/png/1f635.png"></div>

marcozambi

JoshyPHP Brilliant! It worked perfectly, thank you very much for your help and for the tool!

doum

I'm waiting for the release of flarum to migrate my SMF2 forum to flarum so please continue your great job ?

I will test it as soon as possible

-ExotiC-

Well done, nice script. In an alternative version, I did not notice any bugs with the display of HTML ? Working perfectly
But i found 1 weirdness - only 61 users was converted from ~502

marcozambi no passwords are exported. Migrated users will have to ask for a new one

This one makes me cry too ?

marcozambi

-ExotiC- But i found 1 weirdness - only 61 users was converted from ~502

Can you see any commonality among those users? That would help a lot to fix a possible bug.
I'm waiting for the first stable to release an updated importer script ?

-ExotiC-

marcozambi
I do not find any common attributes in old SMF database and in a new Flarum db for imported users ?

P.S.: When i run script i have some note for all users, which was founded to export from SMF:

Notice: Undefined offset: 1 in W:\domains\flarum.local\smf2_to_flarum_alternative.php on line 251
User 000018 - lausin2

And after 100 imported discussions i have

Notice: ob_flush(): failed to flush buffer. No buffer to flush in W:\domains\flarum.local\smf2_to_flarum_alternative.php on line 533

How I can rerun it and clear db of flarum for test the script again? Another clean install of flarum without any entry in DB? Or i can just delete all tuples in DB from 'users' table, 'users_discussions', 'user_groups', 'discussions' and 'posts'? And just reopen the script? Thanks.
Sorry for my English.

-ExotiC-

marcozambi I make screenshot of my old SMF database, maybe I missed something.
Orange highlighted users which have been successfully imported to Flarum database.

WARNING! 7013x751 screenshot size ?
http://i.imgur.com/PP0kCYg.jpg

-ExotiC-

@marcozambi So you have any idea what the problem could be? I still can not find a flaw in the code ?

acseven

-ExotiC- @marcozambi
I'm also trying to use this script to test the migration of a SMF board to Flarum and came across this issue (and a bunch of others). Although I can't say why, the user list is being limited by the last part of the query (the WHERE part):

WHERE 
			".$table_prefix."attachments.attachment_type = 0

If you remove that, it will at least select every member instead of just a few (in my case ~100 in a little over 10000).

On other issues:

I don't think anyone has mentioned importing attachments other than images using flagrow/upload, the replacement for flagrow/flarum-ext-image-upload which the script is using. I have no idea on how to approach this, so if anyone has any pointers I'd be welcome to try.
This line makes no sense: $src_dir = $attachments_dir[$idx_src_dir];and I replaced it by $src_dir = $attachments_dir[0]; and along with removing the WHEREstatement I mentioned earlier makes the import output clean and tidy. But making that change leaves me thinking what is the $idx_src_dirvariable for.

That said, so far I've imported Users, Categories (boards/sub-boards) and Posts. I'm going to check on these to see if everything is passing on as it should or not, and later on I'll try the attachments part.

Edit: Posts imported more or less ok, but it timed-out and I'm yet to verify what's missing.

0E800

acseven Is the SMF bbcode, preserved in the posts? Links? Images?

acseven

EvelinaAdamovi

I'm using the alternative version of the script, and here's the changes I'm using - note that I have not tested importing attachments (i.e. images):

acseven Commented this out:

// Replace multiple (one ore more) line breaks with a single one. $text = preg_replace('/[\r\n]{2,}/s', "\n", $text);

acseven Removed this from the User query:

WHERE ".$table_prefix."attachments.attachment_type = 0

acseven

Replaced:

$src_dir = $attachments_dir[$idx_src_dir];

by this

$src_dir = $attachments_dir[0];

« Previous Page Next Page »