SMF2 to Flarum migration script - alpha 3

sijad

let me know if I can help

sijad

marcozambi did you tried BBCode extension?
those BBCode you did mentioned will be available using it, I guess

see: https://github.com/flarum/flarum-ext-bbcode/blob/master/src/Listener/FormatBBCode.php#L40

marcozambi

sijad Yes, and most of them work, but not the quote nor the links...

marcozambi

Thanks all for your answers.
The problem here is that because we are importing by writing straight to the flarum db, I need to "translate" on the fly the SMF bbcode into the real format used by flarum to store data in the db itself.

Example:
- Original data in SMF: [b]Text in bold[/b]
- Translation in Markdown: **Text in bold**

If I save **Text in bold** in the db, flarum shows something like *Te

Now, of course we try to look behind the curtains, and if I create a post and enter **Text in bold** as text, than post it, clearly this gets "translated" and saved as:

I need to be able to do a proper conversion of bbcode into flarum "lingo" just before saving everything into the db, otherwise the text is simply displayed wrongly.

Do you see my point?

P.S. I have, just to avoid problems, disabled the bbcode extension but left enabled the Markdown (v0.1.0-beta.5) one.
I'm running the latest available beta of flarum (v0.1.0-beta.5).

JoshyPHP

marcozambi If we have to pick one as most important, I would go with [quote] bbcode and nested [quote]. Are they supported at all in flarum?

They are and they can be nested up to 10 levels deep by default. Flarum's quotes don't carry as much information as SMF's. They could be changed via an extension to resemble SMF's more, but I'm not sure that's worth the effort.

luceos why not rewrite to markdown?

It's simpler and less likely to introduce inconsistencies if you the original text as-is. Since Flarum supports BBCodes it should be easier too.

marcozambi Now, of course we try to look behind the curtains, and if I create a post and enter Text in bold as text, than post it, clearly this gets "translated" and saved as:

<r><p><STRONG><s>**</s>Text in bold<e>**</e></STRONG></p></r>

That's XML, generated by the text formatting library that handles BBCodes/Markdown/etc... I'm its author and I keep an eye on Flarum to see how it's used and anticipate users' needs. I would definitely advise against trying to generate the XML directly. If you make any mistake or if there's a special case you didn't foresee you risk making it invalid, and thus unusable.

Instead, grab the original text from SMF (the same text a user would get when editing the post) and use Flarum to parse it. I don't know how you're supposed to get a hold of Flarum's Formatter object from outside the application but I assume you can bootstrap a minimal app and call $this->app->make('flarum.formatter'). You'll have to ask the Flarum devs about it.

If you use Flarum's Formatter object then it will automatically handle YouTube links if a YouTube extension is installed. You may also want to create an extension that adds the missing BBCodes from SMF to Flarum. The library was designed to be able to parse BBCodes from basically any forum software with high fidelity, so it's up to you how close you want your new Flarum to be in terms of BBCodes and other markup compared to SMF. I'm subscribed to this thread so if you have any question about markup, feel free to post them here.

If there's enough demand for it, I may consider publishing a Flarum extension that adds the missing BBCodes and markup. That would require a bunch of research and development so it's unlikely at the moment.

JoshyPHP

marcozambi some problems on translating SMF bbcode correctly

For what it's worth I used to have a TextFormatter bundle for SMF2.1 until I removed it a year or so ago. You may find the missing BBCodes there.

https://github.com/s9e/TextFormatter/blob/c749b84623aa72e8ae033fab2a4980f6aee0f832/src/Configurator/Bundles/S18.php

marcozambi

First post updated, we are on GitHub as of today!
https://github.com/ItalianSpaceAstronauticsAssociation/smf2_to_flarum

wolfman

Thanks for the script! ?

I run an old SMF1 forum which i want to migrate to Flarum - or at least like @marcozambi test Flarum with the original content from SMF1. This script seems to be a good starting point for an SMF1-migration script. I will look at this in a couple of days.

marcozambi

wolfman In the next few days we're going to release further updates. We are currently working on:
- Detecting and converting youtube links into embedded videos (this will work if you have Mediaembed extension installed)
- Smilies conversion to http://emoji.codes/ standard

Later we plan to fix the avatar export, and most probably the image attachments.

wolfman

wolfman I run an old SMF1 forum which i want to migrate to Flarum - or at least like @marcozambi test Flarum with the original content from SMF1. This script seems to be a good starting point for an SMF1-migration script. I will look at this in a couple of days.

Well, couple of days turned into almost 2000 days! But better late then never i guess! 😉 So i finally migrated my SMF1 forum this morning with about 25k topics/336k posts to Flarum and everything went fine so far!

Instead of using the script from @marcozambi as initially intended, i went with the script provided by SHC (see also https://discuss.flarum.org/d/7290-migration-for-a-very-large-smf-forum and https://github.com/sriharshachilakapati/JGO-Flarum-Migration) as the basis for my migration script as it better fitted my needs.

I added some additional functionality to the Script like migrating avatars or importing file-attachments to FoF Upload via Flarum-API, converting internal links, replacing Smileys with Emojis or migrating boards to secondary tags. While some adjustments might be of wider use, others are oddly specific for my migration!

However, if somebody is still running a SMF-Forum and wants to migrate to Flarum (highly recommended!), this might be a good starting point for your migration: https://github.com/wolfgang-demeter/smf2flarum
It will however not run out of the box for you!

I migrated a SMF1 Forum. I do not know if this will work or is compatible with SMF2!

meetdilip

What about polls ?

marcozambi

meetdilip well, the problem is that at the moment afaik flarum does not offer the polls functionality nor in the core neither as extension, so we have literaly no way to port it.
We'll have to wait for that, or start to develop an extension on our own, but we personally have no plan for that.

acseven

marcozambi just a heads up on polls ? up: https://discuss.flarum.org/d/5310-reflar-revolutionizing-flarum/7

marcozambi

JoshyPHP Thank you for your reply.
I managed to understand how to translate most of the SMF bbcode into flarum XML, and I can migrate correctly almost everythig with the script we released yesterday.
It makes a lot of sense to not re-invent the wheel though, so I'll definitively try to update it in the next few days to make use of your beautiful library.

marcozambi

The script has been updated with basic avatar export and an alternative version making use of JoshyPHP s9e\TextFormatter tool.

marcozambi

JoshyPHP Hi there! I'm working again on the code for a better Board-to-Tag conversion, and I decided to abandon the "do-it-yourself" bbcode to XML translation and go fully with your fantastic library s9e/TextFormatter.

I'm using the Forum bundle at the moment, with excellent results, but I noticed that for some reason the emoticons/emoji are not converted.
I tried an experiment, doing a small quick conversion using, instead of the Forum bundle, only the emoji Plugin.

If I try this code

$configurator = new s9e\TextFormatter\Configurator;
$configurator->Emoji->useEmojiOne();
extract($configurator->finalize());

they are converted perfectly.

I was wondering, is there a way (I tried all sort of things) to tweak some settings in the Forum bundle,to set explicitly for example the usage of Emoji->useEmojiOne(); in the Bundle itself?

You did a superb job with this library, but if I may suggest you an improvement that would immensely help potential users,, some extra usage about "mixing" Bundles with Configurator examples would be great!

Thanks in advance!

marcozambi

Added preliminary image attachments export, and some bugfixing.

JoshyPHP

The Emoji plugin was only added recently to the Forum bundle, it's possible your version doesn't include it. Bundles are read-only snapshots of a particular configuration. It's not possible to modify them.

However, you can get a configured instance of s9e\TextFormatter\Configurator that you can use to generate your own instance of the parser and render. You can also save them as your own bundle if you want to reuse them more easily. Or you may want to just serialize the parser/renderer and reuse those instead, as Flarum does. That's the preferred method.

$configurator = s9e\TextFormatter\Configurator\Bundles\Forum::getConfigurator();
$configurator->Emoji->addAlias('x.x', '😵');
$configurator->saveBundle('MyBundle', '/tmp/MyBundle.php');

include '/tmp/MyBundle.php';

$text = '[center]x.x[/center]';
$xml  = MyBundle::parse($text);
$html = MyBundle::render($xml);

die("$html\n");

<div style="text-align:center"><img alt="x.x" class="emoji" draggable="false" width="16" height="16" src="//cdn.jsdelivr.net/emojione/assets/png/1f635.png"></div>

marcozambi

JoshyPHP Brilliant! It worked perfectly, thank you very much for your help and for the tool!

doum

I'm waiting for the release of flarum to migrate my SMF2 forum to flarum so please continue your great job ?

I will test it as soon as possible