SMF2 to Flarum migration script - alpha 3

marcozambi

smf2_to_flarum

Migration script for SMF2 forum to Flarum, partially based on the phpbb_to_flarum script by robrotheram, VIRUXE, Reflic https://github.com/robrotheram/phpbb_to_flarum

Where to download it

You find the script on GitHub
https://github.com/ItalianSpaceAstronauticsAssociation/smf2_to_flarum

Description

The Script exports and migrates your SMF2 (possibly SMF1.x) forum to Flarum. It supports:
- a DB to DB migration,
- a DB to ASCII SQL file export.

These two functionalities can be performed together or as alternatives.

Flarum is still in beta testing, therefore only some of the typical web forum features are available, and what is now working can break anytime (including this script, tailored for flarum ver. 0.1.0-beta.5).
At this moment smf2_to_flarum only supports migration of:
- Users and their avatars (no passwords are exported. Migrated users will have to ask for a new one).
- Boards and sub boards (Categories are intentionally ignored, and boards deeper than 2nd level are added in Flarum as extra Tags)
- Topics and messages (with some bbcode to flarum-markdown translation)
- Attachments (partial, only images)

Changelog

2016-08-28: Version 0.3 Alpha
- Added preliminary image attachments export
- General bugfix.
2016-08-27: Version 0.2 Alpha
- Added preliminary avatars export
- Added alternative script version, based on s9e\TextFormatter library (http://s9etextformatter.readthedocs.io).
- General bugfix.
2016-08-26: Version 0.1 Alpha
- First commit (for flarum ver. 0.1.0-beta.5)

Known issues / Future features

Only SMF/SMF2 image attachments are exported. While SMF supports all kind of attachments, Flarum only allows image attachments and provided that flagrow-image-upload extension has been installed (see https://discuss.flarum.org/d/1836-flagrow-image-uploader-for-flarum-forum-messages)
Private Messages are not exported yet, since this feature is not yet supported by Flarum.
Smilies are not migrated.

Installation

Install a vanilla Flarum, with empty database.
Copy smf2_to_flarum.php (or smf2_to_flarum_alternative.php) and smf2_to_flarum_settings.php in your SMF forum root.
Customise smf2_to_flarum_settings.php script settings with the correct username/password for your SMF2 and (optionally) Flarum DB, and enable/disable the export settings (all enabled by default).
Run the script as http://www.yoursmf2website.xyz/smf2_to_flarum.php

Alternative version

The alternative version of the script (smf2_to_flarum_alternative.php) needs the following tools to be installed on the same server where SMF2 is installed:
- Composer: https://getcomposer.org/download/
- s9etextFormatter: http://s9etextformatter.readthedocs.io/Getting_started/Installation/ (should be in the script's dir)
Rever to the respective websites for install instructions.

Avatars

The script saves users' avatars in a directory named "avatars".
From there the avatars shall be manually copied into flarum "assets/avatars" directory.

Attachments

Attachment management is not part of flarum core. In order to have attachments functionality in flarum the "Image Upload" extension has to be installed and enabled (see https://discuss.flarum.org/d/1836-flagrow-image-uploader-for-flarum-forum-messages).

The migration script exports only image attachments, since no other attachment type is supported at the moment. Files are extracted from single or multiple directory attachment system of SMF/SMF2 then stored in a directory named "images".

From this directory the attachments shall be manually copied in "assets/images" directory of your flarum installation.

Help us

Feel free to contribute to this project!

About us

Hello, my name is Marco and I'm part of the technical board of forumastronautico.it, the largest web forum in Italian dedicated to space enthusiasts. Our forum has over 1300 users and 270000 messages. We are quite happy with SMF itself, but as many long standing web forum platforms, it's starting to fall behind in terms of feature, responsiveness, web2.0-compliance.
We started to look for the "next thing" and flarum looks really really amazing. In order to test Flarum funcionalities in a proper way we decided to migrate our entire forum to Flarum, in a special test environment. This script is the result of the need to move our users, boards hierarchy and topics/messages to the new platform.

luceos

marcozambi let us know if you're running into any issues, we'll gladly support and contribute.

JoshyPHP

marcozambi some problems on translating SMF bbcode correctly

For what it's worth I used to have a TextFormatter bundle for SMF2.1 until I removed it a year or so ago. You may find the missing BBCodes there.

https://github.com/s9e/TextFormatter/blob/c749b84623aa72e8ae033fab2a4980f6aee0f832/src/Configurator/Bundles/S18.php

-ExotiC-

Well done, nice script. In an alternative version, I did not notice any bugs with the display of HTML ? Working perfectly
But i found 1 weirdness - only 61 users was converted from ~502

marcozambi no passwords are exported. Migrated users will have to ask for a new one

This one makes me cry too ?

marcozambi

luceos Thank you very much!
It would be immensely helpful to understand whether is there a list of official "flarum bbcode" (term might be inappropriate though) for [quote], [url], [code], and so forth.
In robotheram script we can see some example, but unfortunately phpbb bbcode is slightly different from smf one, and the adaptations we tried do not seem to work.

If we have to pick one as most important, I would go with [quote] bbcode and nested [quote]. Are they supported at all in flarum?

luceos

marcozambi why not rewrite to markdown?

In case of that bbcode check this thread and otherwise mention the author if you run into issues.

sijad

marcozambi did you tried BBCode extension?
those BBCode you did mentioned will be available using it, I guess

see: https://github.com/flarum/flarum-ext-bbcode/blob/master/src/Listener/FormatBBCode.php#L40

JoshyPHP

marcozambi If we have to pick one as most important, I would go with [quote] bbcode and nested [quote]. Are they supported at all in flarum?

They are and they can be nested up to 10 levels deep by default. Flarum's quotes don't carry as much information as SMF's. They could be changed via an extension to resemble SMF's more, but I'm not sure that's worth the effort.

luceos why not rewrite to markdown?

It's simpler and less likely to introduce inconsistencies if you the original text as-is. Since Flarum supports BBCodes it should be easier too.

marcozambi Now, of course we try to look behind the curtains, and if I create a post and enter Text in bold as text, than post it, clearly this gets "translated" and saved as:

<r><p><STRONG><s>**</s>Text in bold<e>**</e></STRONG></p></r>

That's XML, generated by the text formatting library that handles BBCodes/Markdown/etc... I'm its author and I keep an eye on Flarum to see how it's used and anticipate users' needs. I would definitely advise against trying to generate the XML directly. If you make any mistake or if there's a special case you didn't foresee you risk making it invalid, and thus unusable.

Instead, grab the original text from SMF (the same text a user would get when editing the post) and use Flarum to parse it. I don't know how you're supposed to get a hold of Flarum's Formatter object from outside the application but I assume you can bootstrap a minimal app and call $this->app->make('flarum.formatter'). You'll have to ask the Flarum devs about it.

If you use Flarum's Formatter object then it will automatically handle YouTube links if a YouTube extension is installed. You may also want to create an extension that adds the missing BBCodes from SMF to Flarum. The library was designed to be able to parse BBCodes from basically any forum software with high fidelity, so it's up to you how close you want your new Flarum to be in terms of BBCodes and other markup compared to SMF. I'm subscribed to this thread so if you have any question about markup, feel free to post them here.

If there's enough demand for it, I may consider publishing a Flarum extension that adds the missing BBCodes and markup. That would require a bunch of research and development so it's unlikely at the moment.

sijad

let me know if I can help

marcozambi

sijad Yes, and most of them work, but not the quote nor the links...

marcozambi

Thanks all for your answers.
The problem here is that because we are importing by writing straight to the flarum db, I need to "translate" on the fly the SMF bbcode into the real format used by flarum to store data in the db itself.

Example:
- Original data in SMF: [b]Text in bold[/b]
- Translation in Markdown: **Text in bold**

If I save **Text in bold** in the db, flarum shows something like *Te

Now, of course we try to look behind the curtains, and if I create a post and enter **Text in bold** as text, than post it, clearly this gets "translated" and saved as:

I need to be able to do a proper conversion of bbcode into flarum "lingo" just before saving everything into the db, otherwise the text is simply displayed wrongly.

Do you see my point?

P.S. I have, just to avoid problems, disabled the bbcode extension but left enabled the Markdown (v0.1.0-beta.5) one.
I'm running the latest available beta of flarum (v0.1.0-beta.5).

marcozambi

First post updated, we are on GitHub as of today!
https://github.com/ItalianSpaceAstronauticsAssociation/smf2_to_flarum

wolfman

Thanks for the script! ?

I run an old SMF1 forum which i want to migrate to Flarum - or at least like @marcozambi test Flarum with the original content from SMF1. This script seems to be a good starting point for an SMF1-migration script. I will look at this in a couple of days.

marcozambi

wolfman In the next few days we're going to release further updates. We are currently working on:
- Detecting and converting youtube links into embedded videos (this will work if you have Mediaembed extension installed)
- Smilies conversion to http://emoji.codes/ standard

Later we plan to fix the avatar export, and most probably the image attachments.

wolfman

wolfman I run an old SMF1 forum which i want to migrate to Flarum - or at least like @marcozambi test Flarum with the original content from SMF1. This script seems to be a good starting point for an SMF1-migration script. I will look at this in a couple of days.

Well, couple of days turned into almost 2000 days! But better late then never i guess! 😉 So i finally migrated my SMF1 forum this morning with about 25k topics/336k posts to Flarum and everything went fine so far!

Instead of using the script from @marcozambi as initially intended, i went with the script provided by SHC (see also https://discuss.flarum.org/d/7290-migration-for-a-very-large-smf-forum and https://github.com/sriharshachilakapati/JGO-Flarum-Migration) as the basis for my migration script as it better fitted my needs.

I added some additional functionality to the Script like migrating avatars or importing file-attachments to FoF Upload via Flarum-API, converting internal links, replacing Smileys with Emojis or migrating boards to secondary tags. While some adjustments might be of wider use, others are oddly specific for my migration!

However, if somebody is still running a SMF-Forum and wants to migrate to Flarum (highly recommended!), this might be a good starting point for your migration: https://github.com/wolfgang-demeter/smf2flarum
It will however not run out of the box for you!

I migrated a SMF1 Forum. I do not know if this will work or is compatible with SMF2!

meetdilip

What about polls ?

marcozambi

meetdilip well, the problem is that at the moment afaik flarum does not offer the polls functionality nor in the core neither as extension, so we have literaly no way to port it.
We'll have to wait for that, or start to develop an extension on our own, but we personally have no plan for that.

acseven

marcozambi just a heads up on polls ? up: https://discuss.flarum.org/d/5310-reflar-revolutionizing-flarum/7

marcozambi

JoshyPHP Thank you for your reply.
I managed to understand how to translate most of the SMF bbcode into flarum XML, and I can migrate correctly almost everythig with the script we released yesterday.
It makes a lot of sense to not re-invent the wheel though, so I'll definitively try to update it in the next few days to make use of your beautiful library.

marcozambi

The script has been updated with basic avatar export and an alternative version making use of JoshyPHP s9e\TextFormatter tool.

marcozambi

JoshyPHP Hi there! I'm working again on the code for a better Board-to-Tag conversion, and I decided to abandon the "do-it-yourself" bbcode to XML translation and go fully with your fantastic library s9e/TextFormatter.

I'm using the Forum bundle at the moment, with excellent results, but I noticed that for some reason the emoticons/emoji are not converted.
I tried an experiment, doing a small quick conversion using, instead of the Forum bundle, only the emoji Plugin.

If I try this code

$configurator = new s9e\TextFormatter\Configurator;
$configurator->Emoji->useEmojiOne();
extract($configurator->finalize());

they are converted perfectly.

I was wondering, is there a way (I tried all sort of things) to tweak some settings in the Forum bundle,to set explicitly for example the usage of Emoji->useEmojiOne(); in the Bundle itself?

You did a superb job with this library, but if I may suggest you an improvement that would immensely help potential users,, some extra usage about "mixing" Bundles with Configurator examples would be great!

Thanks in advance!

marcozambi

Added preliminary image attachments export, and some bugfixing.