Upgrading from an old Beta Version

Mallinger

A warm hello to everyone.

The issue

I have a forum running on v0.1.0-beta.7.2. Now I ask myself: What is the shortest way to upgrade to the current version (Flarum v1.6.0)?

I assume that all steps can be performed with PHP 8.0 installed, but I guess my undertaking is not a simple composer update thing.

I have read this instruction from jordanjay29, so I know that changes in the skeleton were made, but is it just the one from v0.1.0-beta.7 to v0.1.0-beta.8? Should I move to v0.1.0-beta.8 first and then all the way to v1.6.0?

I have disabled all non-core extensions. I guess it's advisable to remove them altogether.

Flarum information

Post your output of php flarum info here. Without this information providing support will take longer!

Local web development

Flarum core 0.1.0-beta.7.2
PHP 7.1.33
Loaded extensions: Core, phpdbg_webhelper, date, libxml, openssl, pcre, sqlite3, zlib, bcmath, bz2, calendar, ctype, curl, dba, dom, hash, fileinfo, filter, ftp, gd, gettext, gmp, SPL, iconv, intl, json, ldap, mbstring, session, standard, odbc, pcntl, mysqlnd, PDO, pdo_dblib, pdo_mysql, PDO_ODBC, pdo_pgsql, pdo_sqlite, pgsql, Phar, posix, pspell, readline, Reflection, mysqli, shmop, SimpleXML, soap, sockets, exif, sysvmsg, sysvsem, sysvshm, tidy, tokenizer, wddx, xml, xmlreader, xmlrpc, xmlwriter, xsl, zip, Zend OPcache
EXT flarum-approval v0.1.0-beta.7
EXT flarum-emoji v0.1.0-beta.6
EXT flarum-english v0.1.0-beta.7
EXT flarum-flags v0.1.0-beta.7
EXT flarum-likes v0.1.0-beta.6
EXT flarum-lock v0.1.0-beta.7
EXT flarum-markdown v0.1.0-beta.5
EXT flarum-mentions v0.1.0-beta.7
EXT flarum-sticky v0.1.0-beta.7
EXT flarum-subscriptions v0.1.0-beta.6
EXT flarum-suspend v0.1.0-beta.7
EXT flarum-tags v0.1.0-beta.8
Base URL: http://localhost
Installation path: /Users/xxx/Sites/flarum_local

Online (not accessible due to PHP version)

Flarum core 0.1.0-beta.7.2
PHP 8.0.26
Loaded extensions: Core, date, libxml, openssl, pcre, zlib, filter, hash, json, pcntl, readline, Reflection, SPL, session, standard, restrictdocroot, bcmath, bz2, calendar, ctype, curl, dba, dom, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv, imap, intl, ldap, exif, mysqlnd, PDO, pgsql, Phar, posix, shmop, SimpleXML, soap, sockets, sodium, sqlite3, sysvmsg, sysvsem, sysvshm, tidy, tokenizer, xml, xmlwriter, xsl, mcrypt, mysqli, pdo_mysql, pdo_pgsql, pdo_sqlite, xmlreader, xmlrpc, zip, apcu, decimal, gnupg, igbinary, imagick, mailparse, msgpack, yaml, mongodb, redis, Zend OPcache

Thank you to all who might help.

Mallinger

clarkwinkelmann

Mallinger since you appear to only be using bundled extensions, I would recommend you create a new Flarum 1.6 installation and just swap the database and copy the avatars.

You can edit the database credentials in config.php to point to a different database after installation, or you can delete all tables in the fresh installation and import your existing tables there. The next time you access Flarum you will be asked for the database password and Flarum will automatically update the database structure to the latest version.

Since you only have core extensions, I think the only non-database files you will have to copy are in /assets/avatars. You can copy the entire folder to the new Flarum in the public/assets/avatars folder.

Make sure to keep a backup of the old installation in case something goes wrong!

Mallinger

Hello Clark

Thank you for your fast response.

clarkwinkelmann since you appear to only be using bundled extensions, I would recommend you create a new Flarum 1.6 installation and just swap the database and copy the avatars.

I'm using some other extensions and plan to use them further if possible, I've just deaktivated for the time being. But I understand that these will not be compatible so I have to see what I can use.

Regarding the database I expect that the contents will need some conversion as the syntax e.g. for quotes and mentions might have changed.

clarkwinkelmann The next time you access Flarum you will be asked for the database password and Flarum will automatically update the database structure to the latest version.

This sounds phantastic, in this case my upgrade should be a lot easier than expected. Does the same hold true for contents provided by extensions like e.g. Flagrow Byōbu and Flagrow Upload?

Thanks in advance.

Mallinger

clarkwinkelmann

Mallinger This sounds phantastic, in this case my upgrade should be a lot easier than expected. Does the same hold true for contents provided by extensions like e.g. Flagrow Byōbu and Flagrow Upload?

Yes, all extensions use the database migration API and should be able to update from any previous version directly to the latest.

The only issue I know of is when it comes to extensions that were moved from Flagrow to FriendsOFlarum. The README of those extensions explain which version of the Flagrow extension you should be running before switching to the FriendsOfFlarum version. If your version of the Flagrow extension is too old you will have to update to an intermediate Flarum beta version that still supports the last Flagrow version before you can switch to the FriendsOfFlarum version compatible with Flarum 1.x. This is tricky to achieve because Composer will try to update everything to the most recent version so you would need to modify the Composer commands to install a specific version of Flarum and the extensions.

Mallinger

clarkwinkelmann The only issue I know of is when it comes to extensions that were moved from Flagrow to FriendsOFlarum. The README of those extensions explain which version of the Flagrow extension you should be running before switching to the FriendsOfFlarum version.

I found the requirement of v0.7.1 for flagrow/upload in order to upgrade to the fof version of upload. flagrow upload v0.7.1 requires flarum v0.1.0-beta.8 to work so this upgrade will be my first step.

I haven't found a comparable hint with fof/byobu however. Do you think that flarum v0.1.0-beta.8 will do here as well?

clarkwinkelmann This is tricky to achieve because Composer will try to update everything to the most recent version so you would need to modify the Composer commands to install a specific version of Flarum and the extensions.

Can I find a guide anywhere on how to install / upgrade to a certain version of Flarum or its extensions?

luceos As there might be many changes to the database I highly recommend running php flarum migrate instead of the web based form that asks for the database password.

Thanks for the advice, I will keep it in mind. Right now I'm doing everything within the terminal, locally first, online later.

luceos

As there might be many changes to the database I highly recommend running php flarum migrate instead of the web based form that asks for the database password. That form can time out, causing partially executed migrations. So please do use the commandline option with php flarum migrate.

hrvoje_hr

luceos I can confirm from my own experience this is the best way

https://discuss.flarum.org/d/31036-the-best-way-to-upgrade-from-beta-013-to-the-most-recent-14

luceos Honestly I would do a clean install of the latest version. Then replace the database with your own and run php flarum migrate. Copy over the assets directory into the public directory and clear cache php flarum cache:clear. Then one-by-one install extensions you need (or had, make sure to install successors though).

This is the advice I followed and it worked flawlessly

luceos

Mallinger I found the requirement of v0.7.1 for flagrow/upload in order to upgrade to the fof version of upload. flagrow upload v0.7.1 requires flarum v0.1.0-beta.8 to work so this upgrade will be my first step.

Flagrow (which no longer exists as a team) Upload is superceded by FoF Upload. The best way to see if your old extensions survived the update to stable is to visit https://extiverse.com. See for instance https://extiverse.com/extension/fof/upload

and yes, use a new install, see what hrvoje_hr said.

Mallinger

luceos clarkwinkelmann

In the meanwhile I followed my plan. The first problem arises in the migration process.

Invoking php flarum migrate on the command line produces the following output:

Migrating Flarum...
Migrated: 2018_01_11_093900_change_access_tokens_columns
Migrated: 2018_01_11_094000_change_access_tokens_add_foreign_keys
Migrated: 2018_01_11_095000_change_api_keys_columns
Migrated: 2018_01_11_101800_rename_auth_tokens_to_registration_tokens
Migrated: 2018_01_11_102000_change_registration_tokens_rename_id_to_token
Migrated: 2018_01_11_102100_change_registration_tokens_created_at_to_datetime
Migrated: 2018_01_11_120604_change_posts_table_to_innodb
Migrated: 2018_01_11_155200_change_discussions_rename_columns
Migrated: 2018_01_11_155300_change_discussions_add_foreign_keys
Migrated: 2018_01_15_071700_rename_users_discussions_to_discussion_user
Migrated: 2018_01_15_071800_change_discussion_user_rename_columns
Migrated: 2018_01_15_071900_change_discussion_user_add_foreign_keys
Migrated: 2018_01_15_072600_change_email_tokens_rename_id_to_token
Migrated: 2018_01_15_072700_change_email_tokens_add_foreign_keys
Migrated: 2018_01_15_072800_change_email_tokens_created_at_to_datetime
Migrated: 2018_01_18_130400_rename_permissions_to_group_permission
Migrated: 2018_01_18_130500_change_group_permission_add_foreign_keys
Migrated: 2018_01_18_130600_rename_users_groups_to_group_user
Migrated: 2018_01_18_130700_change_group_user_add_foreign_keys
Migrated: 2018_01_18_133000_change_notifications_columns
Migrated: 2018_01_18_133100_change_notifications_add_foreign_keys
Migrated: 2018_01_18_134400_change_password_tokens_rename_id_to_token
Migrated: 2018_01_18_134500_change_password_tokens_add_foreign_keys
Migrated: 2018_01_18_134600_change_password_tokens_created_at_to_datetime
Migrated: 2018_01_18_135000_change_posts_rename_columns

In Connection.php line 712:
                                                                               
  SQLSTATE[HY000]: General error: 1215 Cannot add foreign key constraint (SQL  
  : alter table `posts` add constraint `posts_user_id_foreign` foreign key (`  
  user_id`) references `users` (`id`) on delete set null)

In Exception.php line 18:
                                                                          
  SQLSTATE[HY000]: General error: 1215 Cannot add foreign key constraint

In PDOStatement.php line 117:
                                                                          
  SQLSTATE[HY000]: General error: 1215 Cannot add foreign key constraint  
                                                                          

migrate

So the migration process has not finished correctly and reloading my forum start page still shows the "Update Flarum" page with the orange Flarum logo at the top.

clarkwinkelmann

Actually, looking at the FoF Upload source code, the migrations seem to be identical in 0.5, 0.6 and 0.7. So it should be fine to do a clean install with FriendsOfFlarum Upload and import the database from any of those versions.

Only if you have Upload version 0.4 or lower you will need to update to an intermediate version. Another solution to avoid having to update to an intermediate version would be to manually apply the changes to the database by looking at the migration code and applying them with SQL commands. I can help provide those SQL commands if needed.

Mallinger

clarkwinkelmann Actually, looking at the FoF Upload source code, the migrations seem to be identical in 0.5, 0.6 and 0.7. So it should be fine to do a clean install with FriendsOfFlarum Upload and import the database from any of those versions.

Great. Can you provide any insight into the requirements of Byobu for the update to the FoF version?

clarkwinkelmann

Mallinger Byobu is probably safe to copy to the latest version.

The database structure appears to be unchanged between Byobu 0.1.0-beta.18 and 0.2.2, the latest Flagrow version. 0.3 is the first FriendsOfFlarum version.

If you are running Byobu 0.1.0-beta.5 to 0.1.0-beta.17 there is one migration to apply, it's the one that set is_private to true for every private discussion in the database following the introduction of that feature in Flarum. You might be able to do this completely manually, or we can write an SQL to do it. Until this is done everyone will be able to see all existing private discussions on the forum.

If you are running Byobu 0.1.0-beta.4 or older, you will need to apply the migration that allows groups to be part of discussions.

Please share which version of the extension you are currently running if you need help with other extensions.

Mallinger

clarkwinkelmann

Thanks Clark

My Upload is version 0.6.0
My Byobu is version 0.1.1
My posts table has the is_private field and it is set to true for the Byobu posts.

So I should be able to switch to the FoF versions of Upload and Byobu without issues? That's:

Creating a fresh Flarum v1.6 installation only with the core extensions,
Copying assets etc. into the public folder,
Point to my database in config.php,
Migrate via php flarum migrate on the command line,
Clear cache via php flarum cache:clear,
Install Byobu and Upload via composer require fof/byobu:"*" and composer require fof/upload

Did I forget anything?

clarkwinkelmann

Mallinger yep, that sounds good! Then look around the new forum to check if anything appears broken.

PS: there is a is_private column on both posts and discussions. It's the one on discussions that was added later on. I don't remember when exactly it was introduced in Flarum but if you have version 0.1.1 of the extension it was probably already there in that Flarum version.

luceos

Mallinger if your own plan implies upgrading one version at a time, I think you will face so many troubles. As recommended by others and myself, install a clean v1.6 (latest) Flarum, no extensions, copy over your database, run php flarum migrate, then start installing the old extensions (or their replacements) and keep running php flarum migrate one by one. It sounds like a shortcut, but it isn't.

Mallinger

luceos if your own plan implies upgrading one version at a time, I think you will face so many troubles.

That's not what I did. I installed a fresh Flarum v1.6.2 without third party extensions and connected it to my old database (which includes tables for my third party extensions of course).

The migration process got interupted at this stage:

Cannot add foreign key constraint (SQL  
  : alter table `posts` add constraint `posts_user_id_foreign` foreign key (`  
  user_id`) references `users` (`id`) on delete set null)

This, it seems to me, is not related to third party extensions.

But I noticed something else: In my old database my tables where of the type MyISAM whereas the new installation came with the InnoDB type for all tables. So I tried to convert all my tables to InnoDB. However, some tables could not be converted due to wrong default values for the created_at field: auth_tokens, email_tokens, flagrow_files, password_tokens, recipients. In the case of flagrow_file_downloads it was the downloaded_at field.

In these cases I had to change the affected field type to datetime with the dafult value of NULL, e.g.

ALTER TABLE `email_tokens` CHANGE `created_at` `created_at` DATETIME NULL DEFAULT NULL;

After that I could change these table types to InnoDB too.

Now the migration process went further including adding foreign keys to the posts table but it stopped at another point:

In Connection.php line 712:
                                                                               
  SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '2-vi  
  ewForum' for key 'PRIMARY' (SQL: update `group_permission` set `permission`  
   = REPLACE(permission,  'viewDiscussions', 'viewForum') where `permission`   
  LIKE %viewDiscussions)

In Exception.php line 18:
                                                                               
  SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '2-vi  
  ewForum' for key 'PRIMARY'

In PDOStatement.php line 117:
                                                                               
  SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '2-vi  
  ewForum' for key 'PRIMARY'

Unfortunately I don't know what to do here. Strange enough, the table permissions from the old forum database is now removed but the table group_permission as to be expected in the new database structure has not been created yet either.

How could I solve that step (I have a database copy from the previous step)?

clarkwinkelmann

Mallinger in earlier database versions there were a lot of foreign keys and unique indexes missing. When updating, those constraints will be added but if you have invalid data that got into the tables at some point Flarum will not always be able to automatically correct it.

For the FOREIGN KEY errors, it looks like the problem was just that the table was not InnoDB. It seems like you got that fixed. It's also possible that FOREIGN KEY errors occur if for example a user_id column has a value for a user that no longer exists in the users table.

For the duplicate entry for permissions, you should be able to go to your permissions table before starting the migration and delete the duplicate values. From the error message it appears you have two different lines that have both group ID 2 and permission name viewForum.

If you want to be extra sure to have a clean database structure, what you could do is after having successfully migrated your old database to the new version with the few manual tweaks you had to do, export the database as data only, and then import that data into a fresh Flarum database. But this will only work after having migrated the old database since there are so many differences between the old tables and new tables. If you do that, make sure you enabled all the extensions in both the migrated and then fresh install so that their tables exist when importing the data-only backup.

Mallinger

Just a short update: I was finally able to perform the migration successfully (until now with just the core extensions).

clarkwinkelmann For the duplicate entry for permissions, you should be able to go to your permissions table before starting the migration and delete the duplicate values. From the error message it appears you have two different lines that have both group ID 2 and permission name viewForum.

In my Beta 7.2 installation of Flarum there were no permissions viewForum. What is now called viewForum in the table group_permission was named viewDiscussions before and it was inside of a table permissions. So during the migration process entries from permissions where translated into entries in group_permission. That's what caused the problem:

Integrity constraint violation: 1062 Duplicate entry '2-vi  
  ewForum' for key 'PRIMARY' (SQL: update `group_permission` set `permission`  
   = REPLACE(permission,  'viewDiscussions', 'viewForum') where `permission`   
  LIKE %viewDiscussions)

Every entry ending in viewDiscussions (LIKE %viewDiscussions) lead to an entry named viewForum and it seems that caused duplicate entries of viewForum. Interestingly this caused problems only in the case of group_id having a value of 2 (guests).

In my case there were 9 entries in the permissions table with a group_id of 2. One of them was viewDiscussions, another 6 ended in viewDiscussions (e.g. tag2.viewDiscussions). Deleting these 6 entries was not enough to have a successfull migration, I had to delete the one with viewDiscussions alone as well. After that the migration went smoothly.

I think it's worth taking a look at the code for the migration process, as it seems to me that the duplicates were created during the migration process. There were definitely no duplicates before the migration.

I will report on the migration of third-party extensions in later posts. However, I would like to take this opportunity to thank everyone who has helped me during the process so far.

clarkwinkelmann

Mallinger good to hear!

Having multiple tagX.viewDiscussions is expected, that's how the tag-scoped permissions are stored, and I think our migration is supposed to rename each one of these while keeping the tagX. prefix. It's strange if we didn't notice this part not working as expected.

Or did you somehow merged a fresh database together with your backup? I think the migration will still try to rename any existing viewDiscussions to viewForum even if there's already a viewForum in the table since it wasn't expected.

Mallinger

clarkwinkelmann Or did you somehow merged a fresh database together with your backup?

No, I replaced the fresh database with my old one and then ran the migration.

clarkwinkelmann I think our migration is supposed to rename each one of these while keeping the tagX. prefix. It's strange if we didn't notice this part not working as expected.

That's what puzzled me as well. And multiple tagX.viewDiscussions entries didn't cause any problems either when the group_id was not equal to 2. I guess it's probably unusual to explicitly allow guests to view a forum (since that's generally the default), and honestly I don't know myself why I chose to do that. Maybe that's why people didn't report this behavior earlier.