Upgrade script

Wadera · Oct 6, 2020

As pointed on discussion jordanjay29
We should upgrade all extensions manually fallowing this guide:
https://discuss.flarum.org/d/9225-the-most-unsettling-end-user-guide-to-composer-for-flarum

If you have multiple of them it will be a lot of manual work, so what about get it automated?

Example fast scripting:

[me@myhost flarum]$ which composer
/usr/local/bin/composer
[me@myhost flarum]$ which php
/usr/local/bin/php
[me@myhost flarum]$ ls
CHANGELOG.md  LICENSE  README.md  composer.json  composer.lock  config.php  extend.php  flarum  nohup.out  public  storage  vendor  update.sh

Version if you have limited memory and want to update one by one:

[me@myhost flarum]$ cat update.sh
#!/bin/bash
extensions=$(cat composer.json | awk '/}/ {found=0} {if(found) print} /require/ {found=1}' | awk -F : '{print $1}' | sed 's/"//g')
echo 'Updating extensions:'
for i in $extensions
do
echo -n 'Updating: '; echo $i;
/usr/local/bin/composer update $i --prefer-dist --no-dev -o --no-suggest
done
echo 'Migrate:'
/usr/local/bin/php flarum migrate
echo 'Clear cache:'
/usr/local/bin/php flarum cache:clear
echo 'Update completed!'
echo

Version if you want to update them all at once:

[me@myhost flarum]$ cat update.sh
#!/bin/bash
extensions=$(cat composer.json | awk '/}/ {found=0} {if(found) print} /require/ {found=1}' | awk -F : '{print $1}' | sed 's/"//g')
echo -n 'Updating extensions: '; echo $extensions
echo
/usr/local/bin/composer update $extensions --prefer-dist --no-dev -o --no-suggest
echo 'Migrate:'
/usr/local/bin/php flarum migrate
echo 'Clear cache:'
/usr/local/bin/php flarum cache:clear
echo 'Update completed!'
echo

How to run it?

[me@myhost flarum]$ chmod +x update.sh
[me@myhost flarum]$ ./update.sh

In theory we can also add it into cron to get autopatch everyday at midnight?
Did I get it right?

clarkwinkelmann · Oct 6, 2020

Wadera well if you're going to update all extensions without testing each update individually, you might as well use the composer update command without a package list. This will be a lot quicker and more memory efficient.

What your script does appears to be what the composer update --root-reqs command would do.

What you always want to do is update, test, ship.

When running commands in production, updating is actually shipping, so there's little room for error.

So if you update one extension at a time and verify everything still works, it's easy to identify what went wrong and revert to the previous version if necessary.

But if you update all extensions at once in production and something goes wrong, you'll have a lot more trouble finding the source of the issue and reverting only what's needed.

That's why we recommend updating extensions one by one, AND TESTING the website still works.

Some extensions might also publish specific update steps that you might want to follow during the update. These usually include the migrate and clear:cache command, which you can safely run after every update even if the extension author didn't specify it. Occasionally there might be additional steps.

If you are in a situation where you have a development server and you put your composer.lock under version control, you can update all at once, verify everything works together, then ship the updated lock file to production. Here updating all packages at once doesn't increase the risks since the commands are not run in production.

If you're going to automate package update, make sure it runs after your nightly database backup and notifies you in case of failure, because if a migration provided by an extension fails for whatever reason, you'll want to know right away to fix/revert your database.

Wadera · Oct 6, 2020

Getting backup (or better - container snapshot) and proper service monitoring is so obvious for me, that I've just forgot to mentioned it

I'm prefer way to get everything updated and if something go wrong - simply restore service from snapshot and then start play with one by one approach and further investigation.

My workflow is:

full CT snapshot
Automated upgrade
Service check
3.1. If everything works fine - done
3.2. If there is something wrong - automating snapshot restore and notify by e-mail

luceos · Oct 7, 2020

A few lessons from experience:

upgrading extensions that apply database changes sometimes cause the inability to downgrade even if you load an older extension version
snapshotting and backups are great, but especially on larger forums, if you have to do a rollback you will lose changes made in the meantime; this can be devastating to community morale
the single best option is to use as much DTAP as you can; development, testing, acceptance and production. Meaning:
- Development, this is where you make changes. This is usually your local computer which runs some kind of webservice, php and mysql virtualised or not (valet, xamp etc). You will run composer command here and manually check everything is in order.
- Testing is where your manager or you test your changes work. This hosting environment resembles your production environment.
- Acceptance is where the client accepts the changes so that they can be deployed.
- Production is where the changes will be made available to whole internet.

You can store most of your flarum forum in GIT for versioning and then move changes down the DTAP pipeline. If it's just you, using the whole pipeline really doesn't make sense and you can scratch TA leaving your local computer and production.

Flarum uses forge (https://forge.laravel.com) to deploy changes made in our forum to production. Forge is a product by the company behind Laravel. It allows us to auto-deploy changes made in our (private) GIT repository to discuss. Here is part of our deployment script:

cd /home/forge/discuss.flarum.org

# stops horizon for queue processing (configured directly on vm)
sudo -n supervisorctl stop discuss:*
# stops kyrne/websockets (configured via forge)
sudo -n supervisorctl stop daemon-<id>:*

# pull latest changes from git
git pull origin master

# proxy running composer install through a custom command
# so that we can capture its output and push it into a private discussion
# as a way to build a deployment archive
php flarum forum:update --discussion=<id>

# in case flarum is installed
if [ -f config.php ]
then
    # flush discussions inside our sandbox testing tag older than 1 year
    php flarum tag:flush sandbox '1 year ago' --force
    # run database changes
    php flarum migrate
    # clear the cache
    php flarum cache:clear
    
fi

# start horizon for queue processing
sudo -n supervisorctl start discuss:*
# start kyrne/websockets
sudo -n supervisorctl start daemon-<id>:*

# restart fpm to clear opcache
echo "" | sudo -S service php7.4-fpm reload

Mmatteocontrini · Oct 22, 2020

luceos It allows us to auto-deploy changes made in our (private) GIT repository to discuss. Here is part of our deployment script:

This is very interesting, I wish we could know more so that it can be used as a "best practice".

For example, what does the repository contain precisely? Is it basically the flarum/flarum repository excluding public/assets and storage/? (But these are probably already excluded by the .gitignore.) And does it include the composer.lock file?

And... If I understand correctly, the command php flarum forum:update --discussion=<id> basically just runs composer install and then posts the output as a new post in an existing discussion, right? Could you share the code, perhaps?

Thanks!

Wadera · Oct 24, 2020

Beta 14 upgrading fast

If you have multiple extensions and you want to upgrade your Flarum - here is few useful commands:

Preparations:

Read carefully official release note: https://discuss.flarum.org/d/25181-flarum-010-beta14-released
Make full backup!!!

Make sure that you are in right place

You below commands need to be run inside your flarum root directory (there where composer.json and config.php reside)

Compatibility check script is on the next post

That's the breaking point

On this point you will need make decision if you want wait little bit longer for more extensions become compatible or remove all incompatible and go ahead with upgrade!

Disable your all extensions

The best way is to simple go to your admin panel and untick all extensions:
https://dev.flarum.local/admin#/extensions

Remove incompatible ones:

Find your last status output (check this post: Wadera ):

[me@myhost flarum]$ ls -larth public/status*

set varilable:

[me@myhost flarum]$ extensionsincompatible=$(grep -B 1 Incompatible public/status-fjgwgiufgiuf_something_unique_fdiugfidwugfis.txt | grep "/")

list incompatible

[me@myhost flarum]$ echo $extensionsincompatible

remove (+ 3 extra mentioned on release note)

[me@myhost flarum]$ composer remove $extensionsincompatible flarum/auth-facebook flarum/auth-github flarum/auth-twitter

Upgrade flarum:

[me@myhost flarum]$ composer require fof/oauth --no-update
[me@myhost flarum]$ composer update --prefer-dist --no-dev -a --with-all-dependencies
[me@myhost flarum]$ php flarum migrate
[me@myhost flarum]$ php flarum cache:clear

Visit your main website. It's possible that you will be ask for your database password to progress DB updates.
You can find it on your config file:

[me@myhost flarum]$ grep password config.php

Your Flarum 14 should be ready to go

Don't forget to remove your extension status file!

[me@myhost flarum]$ rm public/status-*

Wadera · Oct 27, 2020

As I'm one of the people who checking my extensions compatibility everyday to see if I'm reached the point where my flarum is ready to upgrade or I need to wait little bit more - I've gathered some commands and made this simple, reusable script:

Please be aware that you may also try extiverse helper

@luceos created proper tool described here:
https://discuss.flarum.org/d/25246-extiverse-helper

Make sure that you are in right place

You below commands need to be run inside your flarum root directory (there where composer.json and config.php reside)

Checking script: extension-check.sh

#!/bin/bash

# hide location
location=$(date | md5sum | awk '{print $1}')
hostname=$(grep url config.php | awk '{print $3}' | awk -F "'" '{print $2}')
echo `date` > public/status-$location.txt


# Setup extensions list:
extensions=$(cat composer.json | awk '/}/ {found=0} {if(found) print} /require/ {found=1}' | awk -F : '{print $1}' | sed 's/"//g')
res=$(echo "$extensions" | wc -l)
counter=1

echo
echo "Found $res extensions"
echo

echo "Checking..."
echo

# Checking status on extiverse.com
for i in $extensions
do
echo "------------------------------------------------------------" >>  public/status-$location.txt
echo $i >> public/status-$location.txt
curl -s https://extiverse.com/extension/$i | grep -i compa >> public/status-$location.txt
echo "https://extiverse.com/extension/$i">> public/status-$location.txt
echo -n $counter ; echo -n ' / ' ; echo $res ; ((counter++))
done

echo '' >> public/status-$location.txt
echo "------------------------------------------------------------------------------------"  >> public/status-$location.txt
echo '' >> public/status-$location.txt
php flarum info >> public/status-$location.txt

# That's your list:
echo
echo "----------------------------------------------------------------"
echo
echo "Check result:"
#echo
#php flarum info
#echo
#echo "----------------------------------------------------------------"
#echo
#echo You can check it here:
echo "$hostname/status-$location.txt"
echo
echo "More information:"
echo "https://discuss.flarum.org/d/25061-upgrade-script/6"
echo
echo "Don't forgot to remove status file when you check everything!!!"
echo
echo "Command to remove:"
echo "rm public/status-*"
echo

To run:

$ ./extension-check.sh

Example output:

$ ./extension-check.sh

Found 42 extensions

Checking...

1 / 42
2 / 42
3 / 42
4 / 42
5 / 42
6 / 42
7 / 42
8 / 42
9 / 42
10 / 42
11 / 42
12 / 42
13 / 42
14 / 42
15 / 42
16 / 42
17 / 42
18 / 42
19 / 42
20 / 42
21 / 42
22 / 42
23 / 42
24 / 42
25 / 42
26 / 42
27 / 42
28 / 42
29 / 42
30 / 42
31 / 42
32 / 42
33 / 42
34 / 42
35 / 42
36 / 42
37 / 42
38 / 42
39 / 42
40 / 42
41 / 42
42 / 42


----------------------------------------------------------------

Check result:
https://flarum.dev.local/status-08fa67f70fea3326f85fb18c126eb2c03c.txt

More information:
https://discuss.flarum.org/d/25061-upgrade-script/6

Don't forgot to remove status file when you check everything!!!

Command to remove:
rm public/status-*

And file available under pointed address:

Tue Oct 27 21:24:48 UTC 2020
------------------------------------------------------------
askvortsov/flarum-categories
                    Compatible
https://extiverse.com/extension/askvortsov/flarum-categories
------------------------------------------------------------
clarkwinkelmann/flarum-ext-author-change
                    Compatible
<p>Bugfixes and compatibility updates will be published for free as time allows.</p>
https://extiverse.com/extension/clarkwinkelmann/flarum-ext-author-change
------------------------------------------------------------
clarkwinkelmann/flarum-ext-emojionearea
                    Incompatible
https://extiverse.com/extension/clarkwinkelmann/flarum-ext-emojionearea
------------------------------------------------------------
flagrow/upload
https://extiverse.com/extension/flagrow/upload
------------------------------------------------------------
flarum/approval
                    Compatible
https://extiverse.com/extension/flarum/approval
------------------------------------------------------------
flarum/bbcode
                    Incompatible
https://extiverse.com/extension/flarum/bbcode
------------------------------------------------------------
flarum/core
                    Incompatible
https://extiverse.com/extension/flarum/core
------------------------------------------------------------
flarum/emoji
                    Compatible
https://extiverse.com/extension/flarum/emoji
------------------------------------------------------------
flarum/flags
                    Compatible
https://extiverse.com/extension/flarum/flags
------------------------------------------------------------
flarum/lang-english
                    Compatible
https://extiverse.com/extension/flarum/lang-english
------------------------------------------------------------
flarum/likes
                    Compatible
https://extiverse.com/extension/flarum/likes
------------------------------------------------------------
flarum/lock
                    Compatible
https://extiverse.com/extension/flarum/lock
------------------------------------------------------------
flarum/markdown
                    Compatible
https://extiverse.com/extension/flarum/markdown
------------------------------------------------------------
flarum/mentions
                    Compatible
https://extiverse.com/extension/flarum/mentions
------------------------------------------------------------
flarum/statistics
                    Compatible
https://extiverse.com/extension/flarum/statistics
------------------------------------------------------------
flarum/sticky
                    Compatible
https://extiverse.com/extension/flarum/sticky
------------------------------------------------------------
flarum/subscriptions
                    Compatible
https://extiverse.com/extension/flarum/subscriptions
------------------------------------------------------------
flarum/suspend
                    Compatible
https://extiverse.com/extension/flarum/suspend
------------------------------------------------------------
flarum/tags
                    Compatible
https://extiverse.com/extension/flarum/tags
------------------------------------------------------------
fof/best-answer
                    Incompatible
https://extiverse.com/extension/fof/best-answer
------------------------------------------------------------
fof/byobu
                    Incompatible
https://extiverse.com/extension/fof/byobu
------------------------------------------------------------
fof/follow-tags
                    Incompatible
https://extiverse.com/extension/fof/follow-tags
------------------------------------------------------------
fof/linguist
                    Incompatible
<p>To the latest compatible version:</p>
https://extiverse.com/extension/fof/linguist
------------------------------------------------------------
fof/links
                    Incompatible
https://extiverse.com/extension/fof/links
------------------------------------------------------------
fof/merge-discussions
                    Incompatible
https://extiverse.com/extension/fof/merge-discussions
------------------------------------------------------------
fof/nightmode
                    Incompatible
https://extiverse.com/extension/fof/nightmode
------------------------------------------------------------
fof/polls
                    Incompatible
https://extiverse.com/extension/fof/polls
------------------------------------------------------------
fof/profile-image-crop
                    Incompatible
https://extiverse.com/extension/fof/profile-image-crop
------------------------------------------------------------
fof/reactions
                    Incompatible
https://extiverse.com/extension/fof/reactions
------------------------------------------------------------
fof/realtimedate
                    Incompatible
https://extiverse.com/extension/fof/realtimedate
------------------------------------------------------------
fof/recaptcha
                    Incompatible
https://extiverse.com/extension/fof/recaptcha
------------------------------------------------------------
fof/spamblock
                    Incompatible
https://extiverse.com/extension/fof/spamblock
------------------------------------------------------------
fof/split
                    Compatible
https://extiverse.com/extension/fof/split
------------------------------------------------------------
fof/upload
                    Incompatible
https://extiverse.com/extension/fof/upload
------------------------------------------------------------
fof/user-bio
                    Compatible
https://extiverse.com/extension/fof/user-bio
------------------------------------------------------------
fof/user-directory
                    Incompatible
<p>To the latest compatible version:</p>
https://extiverse.com/extension/fof/user-directory
------------------------------------------------------------
kilowhat/flarum-ext-audit-free
                    Compatible
https://extiverse.com/extension/kilowhat/flarum-ext-audit-free
------------------------------------------------------------
kvothe/online-users
                    Incompatible
https://extiverse.com/extension/kvothe/online-users
------------------------------------------------------------
kyrne/shout
                    Compatible
https://extiverse.com/extension/kyrne/shout
------------------------------------------------------------
kyrne/websocket
                    Incompatible
                            <span class="truncate"><p>I definitely recommend this extension a lot!<br></p><p>It achieved a 5-10% improvement compared to flarum/pusher. Beside the bokt/flarum-redis extension I felt an incredible optimization on my serv...</span>
<h2>Extension Compatibility</h2>
https://extiverse.com/extension/kyrne/websocket
------------------------------------------------------------
michaelbelgium/flarum-discussion-views
                    Compatible
https://extiverse.com/extension/michaelbelgium/flarum-discussion-views
------------------------------------------------------------
migratetoflarum/recalculate-meta
                    Incompatible
<p>Compatible with both beta 13 and beta 14.</p>
https://extiverse.com/extension/migratetoflarum/recalculate-meta
------------------------------------------------------------------------------------

Flarum core 0.1.0-beta.13
PHP version: 7.4.11
Loaded extensions: Core, date, libxml, openssl, pcre, sqlite3, zlib, bcmath, calendar, ctype, curl, dom, hash, fileinfo, filter, ftp, gd, gettext, SPL, iconv, intl, json, mbstring, session, standard, mysqlnd, PDO, pdo_mysql, pdo_sqlite, Phar, posix, Reflection, mysqli, SimpleXML, soap, sockets, sodium, exif, tokenizer, xml, xmlreader, xmlwriter, xsl, zip, imagick, ionCube Loader
+----------------------------------+------------------+--------+
| Flarum Extensions                |                  |        |
+----------------------------------+------------------+--------+
| ID                               | Version          | Commit |
+----------------------------------+------------------+--------+
| flarum-approval                  | v0.1.0-beta.13   |        |
| flarum-bbcode                    | v0.1.0-beta.12   |        |
| flarum-emoji                     | v0.1.0-beta.13   |        |
| flarum-lang-english              | v0.1.0-beta.13   |        |
| flarum-flags                     | v0.1.0-beta.13   |        |
| flarum-likes                     | v0.1.0-beta.13   |        |
| flarum-lock                      | v0.1.0-beta.13   |        |
| flarum-markdown                  | v0.1.0-beta.13   |        |
| flarum-mentions                  | v0.1.0-beta.13   |        |
| flarum-statistics                | v0.1.0-beta.13   |        |
| flarum-sticky                    | v0.1.0-beta.13   |        |
| flarum-subscriptions             | v0.1.0-beta.13   |        |
| flarum-suspend                   | v0.1.0-beta.13   |        |
| flarum-tags                      | v0.1.0-beta.13.2 |        |
| fof-user-bio                     | 0.2.0            |        |
| fof-links                        | 0.3.0            |        |
| fof-polls                        | 0.1.2            |        |
| fof-spamblock                    | 0.2.2            |        |
| fof-split                        | 0.4.4            |        |
| fof-follow-tags                  | 0.4.4            |        |
| fof-recaptcha                    | 0.1.2            |        |
| fof-profile-image-crop           | 0.1.2            |        |
| fof-user-directory               | 0.3.4            |        |
| fof-linguist                     | 0.4.3            |        |
| michaelbelgium-discussion-views  | v4.0.1           |        |
| fof-upload                       | 0.10.0           |        |
| fof-merge-discussions            | 0.3.3            |        |
| fof-nightmode                    | 0.5.2            |        |
| fof-reactions                    | 0.3.4            |        |
| kvothe-online-users              | 0.4.0            |        |
| fof-realtimedate                 | 0.1.3            |        |
| kyrne-shout                      | 0.1.19           |        |
| kilowhat-audit-free              | 1.2.1            |        |
| kyrne-websocket                  | 1.0.7            |        |
| clarkwinkelmann-author-change    | 0.2.0            |        |
| fof-byobu                        | 0.5.6            |        |
| askvortsov-categories            | v1.0.4           |        |
| clarkwinkelmann-emojionearea     | 0.2.2            |        |
| fof-best-answer                  | 0.1.11           |        |
| migratetoflarum-recalculate-meta | 0.1.0            |        |
+----------------------------------+------------------+--------+
Base URL: https://flarum.dev.local
Installation path: /home/dev/domains/flarum.dev.local/flarum
Debug mode: off

MikeJones · Oct 27, 2020

Wadera thats awesome! I will have to try this later!

peopleinside · Oct 29, 2020

Wadera for me on Centos command to run is not ./ but sh
and I get this:

extension-check.sh: line 2: $'\r': command not found
extension-check.sh: line 7: $'\r': command not found
extension-check.sh: line 8: $'\r': command not found
extension-check.sh: line 13: $'\r': command not found
extension-check.sh: line 14: $'echo\r': command not found
 extensions
extension-check.sh: line 16: $'echo\r': command not found
extension-check.sh: line 17: $'\r': command not found
Checking...
extension-check.sh: line 19: $'echo\r': command not found
extension-check.sh: line 20: $'\r': command not found
extension-check.sh: line 23: syntax error near unexpected token `$'do\r''
'xtension-check.sh: line 23: `do

Wadera · Oct 29, 2020

If we talking about this commands:

Wadera Remove incompatible ones:

# Find your last status output:
$ ls -larth public/status*
# set varilable:
$ extensionsincompatible=$(grep -B 1 Incompatible public/status-fjgwgiufgiuf_something_unique_fdiugfidwugfis.txt | grep "/")
# list incompatible 
$ echo $extensionsincompatible
# remove (+ 3 extra mentioned on release note)
$ composer remove $extensionsincompatible flarum/auth-facebook flarum/auth-github flarum/auth-twitter

Then: $ is just your command prompt (usually something like: me@myserver /my/location $ and # is commented line

You should understand it as:

To find your last status output please run this command as normal user:

ls -larth public/status*

Then to set varilable please run this command as normal user:

 extensionsincompatible=$(grep -B 1 Incompatible public/status-fjgwgiufgiuf_something_unique_fdiugfidwugfis.txt | grep "/")`

and so one...