Upgrading from an old Beta Version

clarkwinkelmann

Actually, looking at the FoF Upload source code, the migrations seem to be identical in 0.5, 0.6 and 0.7. So it should be fine to do a clean install with FriendsOfFlarum Upload and import the database from any of those versions.

Only if you have Upload version 0.4 or lower you will need to update to an intermediate version. Another solution to avoid having to update to an intermediate version would be to manually apply the changes to the database by looking at the migration code and applying them with SQL commands. I can help provide those SQL commands if needed.

Mallinger

clarkwinkelmann Actually, looking at the FoF Upload source code, the migrations seem to be identical in 0.5, 0.6 and 0.7. So it should be fine to do a clean install with FriendsOfFlarum Upload and import the database from any of those versions.

Great. Can you provide any insight into the requirements of Byobu for the update to the FoF version?

clarkwinkelmann

Mallinger Byobu is probably safe to copy to the latest version.

The database structure appears to be unchanged between Byobu 0.1.0-beta.18 and 0.2.2, the latest Flagrow version. 0.3 is the first FriendsOfFlarum version.

If you are running Byobu 0.1.0-beta.5 to 0.1.0-beta.17 there is one migration to apply, it's the one that set is_private to true for every private discussion in the database following the introduction of that feature in Flarum. You might be able to do this completely manually, or we can write an SQL to do it. Until this is done everyone will be able to see all existing private discussions on the forum.

If you are running Byobu 0.1.0-beta.4 or older, you will need to apply the migration that allows groups to be part of discussions.

Please share which version of the extension you are currently running if you need help with other extensions.

Mallinger

clarkwinkelmann

Thanks Clark

My Upload is version 0.6.0
My Byobu is version 0.1.1
My posts table has the is_private field and it is set to true for the Byobu posts.

So I should be able to switch to the FoF versions of Upload and Byobu without issues? That's:

Creating a fresh Flarum v1.6 installation only with the core extensions,
Copying assets etc. into the public folder,
Point to my database in config.php,
Migrate via php flarum migrate on the command line,
Clear cache via php flarum cache:clear,
Install Byobu and Upload via composer require fof/byobu:"*" and composer require fof/upload

Did I forget anything?

clarkwinkelmann

Mallinger yep, that sounds good! Then look around the new forum to check if anything appears broken.

PS: there is a is_private column on both posts and discussions. It's the one on discussions that was added later on. I don't remember when exactly it was introduced in Flarum but if you have version 0.1.1 of the extension it was probably already there in that Flarum version.

Mallinger

luceos clarkwinkelmann

In the meanwhile I followed my plan. The first problem arises in the migration process.

Invoking php flarum migrate on the command line produces the following output:

Migrating Flarum...
Migrated: 2018_01_11_093900_change_access_tokens_columns
Migrated: 2018_01_11_094000_change_access_tokens_add_foreign_keys
Migrated: 2018_01_11_095000_change_api_keys_columns
Migrated: 2018_01_11_101800_rename_auth_tokens_to_registration_tokens
Migrated: 2018_01_11_102000_change_registration_tokens_rename_id_to_token
Migrated: 2018_01_11_102100_change_registration_tokens_created_at_to_datetime
Migrated: 2018_01_11_120604_change_posts_table_to_innodb
Migrated: 2018_01_11_155200_change_discussions_rename_columns
Migrated: 2018_01_11_155300_change_discussions_add_foreign_keys
Migrated: 2018_01_15_071700_rename_users_discussions_to_discussion_user
Migrated: 2018_01_15_071800_change_discussion_user_rename_columns
Migrated: 2018_01_15_071900_change_discussion_user_add_foreign_keys
Migrated: 2018_01_15_072600_change_email_tokens_rename_id_to_token
Migrated: 2018_01_15_072700_change_email_tokens_add_foreign_keys
Migrated: 2018_01_15_072800_change_email_tokens_created_at_to_datetime
Migrated: 2018_01_18_130400_rename_permissions_to_group_permission
Migrated: 2018_01_18_130500_change_group_permission_add_foreign_keys
Migrated: 2018_01_18_130600_rename_users_groups_to_group_user
Migrated: 2018_01_18_130700_change_group_user_add_foreign_keys
Migrated: 2018_01_18_133000_change_notifications_columns
Migrated: 2018_01_18_133100_change_notifications_add_foreign_keys
Migrated: 2018_01_18_134400_change_password_tokens_rename_id_to_token
Migrated: 2018_01_18_134500_change_password_tokens_add_foreign_keys
Migrated: 2018_01_18_134600_change_password_tokens_created_at_to_datetime
Migrated: 2018_01_18_135000_change_posts_rename_columns

In Connection.php line 712:
                                                                               
  SQLSTATE[HY000]: General error: 1215 Cannot add foreign key constraint (SQL  
  : alter table `posts` add constraint `posts_user_id_foreign` foreign key (`  
  user_id`) references `users` (`id`) on delete set null)

In Exception.php line 18:
                                                                          
  SQLSTATE[HY000]: General error: 1215 Cannot add foreign key constraint

In PDOStatement.php line 117:
                                                                          
  SQLSTATE[HY000]: General error: 1215 Cannot add foreign key constraint  
                                                                          

migrate

So the migration process has not finished correctly and reloading my forum start page still shows the "Update Flarum" page with the orange Flarum logo at the top.

luceos

Mallinger if your own plan implies upgrading one version at a time, I think you will face so many troubles. As recommended by others and myself, install a clean v1.6 (latest) Flarum, no extensions, copy over your database, run php flarum migrate, then start installing the old extensions (or their replacements) and keep running php flarum migrate one by one. It sounds like a shortcut, but it isn't.

Mallinger

luceos if your own plan implies upgrading one version at a time, I think you will face so many troubles.

That's not what I did. I installed a fresh Flarum v1.6.2 without third party extensions and connected it to my old database (which includes tables for my third party extensions of course).

The migration process got interupted at this stage:

Cannot add foreign key constraint (SQL  
  : alter table `posts` add constraint `posts_user_id_foreign` foreign key (`  
  user_id`) references `users` (`id`) on delete set null)

This, it seems to me, is not related to third party extensions.

But I noticed something else: In my old database my tables where of the type MyISAM whereas the new installation came with the InnoDB type for all tables. So I tried to convert all my tables to InnoDB. However, some tables could not be converted due to wrong default values for the created_at field: auth_tokens, email_tokens, flagrow_files, password_tokens, recipients. In the case of flagrow_file_downloads it was the downloaded_at field.

In these cases I had to change the affected field type to datetime with the dafult value of NULL, e.g.

ALTER TABLE `email_tokens` CHANGE `created_at` `created_at` DATETIME NULL DEFAULT NULL;

After that I could change these table types to InnoDB too.

Now the migration process went further including adding foreign keys to the posts table but it stopped at another point:

In Connection.php line 712:
                                                                               
  SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '2-vi  
  ewForum' for key 'PRIMARY' (SQL: update `group_permission` set `permission`  
   = REPLACE(permission,  'viewDiscussions', 'viewForum') where `permission`   
  LIKE %viewDiscussions)

In Exception.php line 18:
                                                                               
  SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '2-vi  
  ewForum' for key 'PRIMARY'

In PDOStatement.php line 117:
                                                                               
  SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '2-vi  
  ewForum' for key 'PRIMARY'

Unfortunately I don't know what to do here. Strange enough, the table permissions from the old forum database is now removed but the table group_permission as to be expected in the new database structure has not been created yet either.

How could I solve that step (I have a database copy from the previous step)?

clarkwinkelmann

Mallinger in earlier database versions there were a lot of foreign keys and unique indexes missing. When updating, those constraints will be added but if you have invalid data that got into the tables at some point Flarum will not always be able to automatically correct it.

For the FOREIGN KEY errors, it looks like the problem was just that the table was not InnoDB. It seems like you got that fixed. It's also possible that FOREIGN KEY errors occur if for example a user_id column has a value for a user that no longer exists in the users table.

For the duplicate entry for permissions, you should be able to go to your permissions table before starting the migration and delete the duplicate values. From the error message it appears you have two different lines that have both group ID 2 and permission name viewForum.

If you want to be extra sure to have a clean database structure, what you could do is after having successfully migrated your old database to the new version with the few manual tweaks you had to do, export the database as data only, and then import that data into a fresh Flarum database. But this will only work after having migrated the old database since there are so many differences between the old tables and new tables. If you do that, make sure you enabled all the extensions in both the migrated and then fresh install so that their tables exist when importing the data-only backup.

Mallinger

Just a short update: I was finally able to perform the migration successfully (until now with just the core extensions).

clarkwinkelmann For the duplicate entry for permissions, you should be able to go to your permissions table before starting the migration and delete the duplicate values. From the error message it appears you have two different lines that have both group ID 2 and permission name viewForum.

In my Beta 7.2 installation of Flarum there were no permissions viewForum. What is now called viewForum in the table group_permission was named viewDiscussions before and it was inside of a table permissions. So during the migration process entries from permissions where translated into entries in group_permission. That's what caused the problem:

Integrity constraint violation: 1062 Duplicate entry '2-vi  
  ewForum' for key 'PRIMARY' (SQL: update `group_permission` set `permission`  
   = REPLACE(permission,  'viewDiscussions', 'viewForum') where `permission`   
  LIKE %viewDiscussions)

Every entry ending in viewDiscussions (LIKE %viewDiscussions) lead to an entry named viewForum and it seems that caused duplicate entries of viewForum. Interestingly this caused problems only in the case of group_id having a value of 2 (guests).

In my case there were 9 entries in the permissions table with a group_id of 2. One of them was viewDiscussions, another 6 ended in viewDiscussions (e.g. tag2.viewDiscussions). Deleting these 6 entries was not enough to have a successfull migration, I had to delete the one with viewDiscussions alone as well. After that the migration went smoothly.

I think it's worth taking a look at the code for the migration process, as it seems to me that the duplicates were created during the migration process. There were definitely no duplicates before the migration.

I will report on the migration of third-party extensions in later posts. However, I would like to take this opportunity to thank everyone who has helped me during the process so far.

clarkwinkelmann

Mallinger good to hear!

Having multiple tagX.viewDiscussions is expected, that's how the tag-scoped permissions are stored, and I think our migration is supposed to rename each one of these while keeping the tagX. prefix. It's strange if we didn't notice this part not working as expected.

Or did you somehow merged a fresh database together with your backup? I think the migration will still try to rename any existing viewDiscussions to viewForum even if there's already a viewForum in the table since it wasn't expected.

Mallinger

clarkwinkelmann Or did you somehow merged a fresh database together with your backup?

No, I replaced the fresh database with my old one and then ran the migration.

clarkwinkelmann I think our migration is supposed to rename each one of these while keeping the tagX. prefix. It's strange if we didn't notice this part not working as expected.

That's what puzzled me as well. And multiple tagX.viewDiscussions entries didn't cause any problems either when the group_id was not equal to 2. I guess it's probably unusual to explicitly allow guests to view a forum (since that's generally the default), and honestly I don't know myself why I chose to do that. Maybe that's why people didn't report this behavior earlier.

Mallinger

@luceos and @clarkwinkelmann

I continued my upgrade process by trying to install the first third-party extensions, all quite common:

fof/pages, fof/links and fof/upload

In all three cases, I received a yellow deprecation notice (likely negligible):

Deprecation Notice: Required parameter $path follows optional parameter $schema in phar:///usr/local/bin/composer/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/UndefinedConstraint.php:53
Deprecation Notice: Required parameter $path follows optional parameter $schema in phar:///usr/local/bin/composer/vendor/justinrainbow/json-schema/src/JsonSchema/Constraints/UndefinedConstraint.php:97

followed by a red error exception:

  [ErrorException]
  Trying to access array offset on value of type null

which terminated the process.

In all cases I ran e.g. composer require fof/pages on the command line from within my flarum directory.
PHP is version 8.0.26, both with Apache 2.0 (phpinfo.php) and on the command line (php -v).
My composer.json is still the one from the fresh v1.6.2 installation.

Has anyone an idea?

clarkwinkelmann

Mallinger the ErrorException was thrown by Composer, or by Flarum?

In either case you can try running the command with -v flag to get more verbose output which will tell us in which file the error happened.

Can you also check composer --version to see whether you have an outdated Composer version?

Mallinger

Thank you Clark.

clarkwinkelmann Can you also check composer --version to see whether you have an outdated Composer version?

Me stupid. On my computer I need to call php /usr/local/composer.phar for composer 2, simple composer still invokes composer 1 which seems to fail with pretty everything including composer --version.

php /usr/local/composer.phar fof/pages etc. did the job beautifully.