Dev Diary: Beta 8

Toby

Last week @Franz, @luceos, @jordanjay29, and I had a meeting to discuss some big-picture aspects of the Flarum project. One of the things we wanted was to be more transparent about development activity and engage the community with our progress. Keeping a handle on the GitHub commits and issues is tedious and it's not always obvious what we're working on, why we're working on it, or that progress is being made at all.

So we're starting this series called the Dev Diary. Each of us core devs will occasionally give an update on our progress, add some insights on the changes we've made, or talk about some nerdy coding stuff. We'll create a new Dev Diary discussion after each release. Replies are disabled for the public – if you wish to comment on a particular post, please start a new discussion.

Without further ado, here is the Dev Diary for the upcoming beta 8 release! We'll do our best to post here regularly and we hope you enjoy reading.

luceos

The last few weeks I have been pushing code into two different branches primarily. The first adding tests to verify the Api controllers are doing what they should. The second is changing important parts of the database scheme to make it more uniform. By using the first branch code inside the second I have been able to confirm that flarum/core is no longer breaking.

There are a few things on my todo for the coming period;

add even more tests to improve our overal code coverage and increase our stability,
finish the database changes, all core extensions have to be modified to comply,
look at the state of the abstract oauth 2 branch and finish it up,
investigate getting the Flarum trademark and/or legal set up to make the trademark happen.

Adding tests increases code coverage. A number signifying how much of your code is covered by tests. With increasing code coverage it will be easier to exclude recurring bugs and give out a guarantee of the stability of Flarum as a whole without a need for human interference or approval.

We've been talking about acquiring a Flarum trademark lately. In order to do so we side stepped into a transparent way of acquiring funds by using Open Collective. The concept of having an Open Collective is to be transparent what money comes in and what it is used for.

The database changes is a task set out for beta 8, combined into one pull request. It will have an affect on any extension using a core class in the backend (php).

Toby

Over the past few weeks I've been working on some major improvements to Flarum's JavaScript tooling and asset pipeline. ??

Webpack

Previously we used a custom Gulp workflow to compile our ES6 source code into something that browsers could understand. This setup worked, but it was pretty unconventional. It also limited our ability to explicitly define a public API for extensions, which would've made backwards compatibility harder going forward.

Now I have almost finished switching us over to Webpack, which is a very popular tool for this kind of thing among the JavaScript community. Adopting common technologies like this is a often good thing because they are tried and tested and a lot of people are familiar with them!

The new Webpack setup is gold-standard. As opposed to converting all of the source code into System.js modules (all accessible to extensions), only the modules we export directly from index.js will be made public. This means we will be able to explicitly define our new public JavaScript API (one of the things I'll be working on next). In the meantime, backwards compatibility with the old style of API consumption remains. As such, beta 8 will require some minor re-tooling of extension JavaScript code, but no massive changes to the code itself.

Folder Structure

This change has also involved making some changes to the JS folder structure. Now there will only be one npm instance in the package root, and you will only have to run npm run build to compile both forum and admin JavaScript files. I am still finalising these changes and am aiming to push them soon.

Travis CI

One of the things we've been meaning to do for a while is to set up a bot that runs on Travis CI to automatically re-compile the JavaScript files whenever new code is pushed to master. This will help us to avoid conflicts when developers run slightly different versions of the JavaScript tooling with slightly different outputs. It will also result in a safer codebase because people will not have the opportunity to hide malicious code in their compiled JS when submitting a pull request.

Asset Manager

Finally, I have been working on some improvements to the asset pipeline, where Flarum collates the JavaScript files from core + extensions and serves them up to the browser. Previously we applied minification to the compiled JavaScript using PHP, but this was slow and made enabling/disabling extensions take a long time. Now, Webpack does this job for us and we will commit the minified output to the GitHub repo so it's ready to be used by Flarum installations straight away!

I have fixed numerous bugs along the way, eliminating some common problems like Flarum showing translation keys instead of the actual English translations, and CSS not recompiling during development when you make some changes to a LESS file.

Additionally, Webpack produces source maps when it compiles the JavaScript. I have added functionality to Flarum's asset manager so it will serve these up as well (if debug mode is on) so that developers can see JavaScript errors in the context of the original, unminified, uncompiled source code. The LESS parser we're using also supports source maps so I was able to enable it for that as well. Very cool!

Up Next

Finalize the new JS folder structure and apply it to all extensions
Merge the flarum/core webpack branch into master
Set up Travis CI JavaScript compilation for all extensions
Clean up asset manager changes and merge into master
Begin work on the foundation of the new JavaScript Extension API

luceos

A quick update from my side.

I've looked at the issue related to MariaDB 10.2. In order to properly test compliance of our code I've updated our tests for travis to test against both MySQL and MariaDB. The conclusion from these tests is that in order to support it we need to move our minimum php requirement to 7.1. With PHP 7.0 being EOL at the end of the year it's a very tempting choice. The discussion is taking place in the issue: flarum/core1211 and the pull request: https://github.com/flarum/core/pull/1440.

In addition to this I had an open question related to the users.preferences column which will be deprecated in beta 8. We decided to move the preferences related to notifications to a separate table and all others will be new columns on the users table itself. I can now continue completing that task. See pull request: https://github.com/flarum/core/pull/1344.

Franz

Allow me to join in on the fun as well. ?

PHP minimum requirements

After stumbling into this discussion several times in the last few months, we finally decided to change our minimum PHP requirement for the upcoming beta (and the first stable release) once more to PHP 7.1. This allows us to upgrade several of our dependencies, such as Laravel. More importantly, it allows us to start officially supporting MariaDB.

PSR-15 middleware

A rather technical topic, but still useful...

With PHP 7.1 out of the way, I just performed the upgrade to version 3.0 of zend-stratigility, another one of our core dependencies. This one is at the very heart of Flarum, and takes care of dispatching all incoming HTTP requests (any "page" you open on a Flarum forum, as well as any data that is fetched or sent in the background) through a stack of so-called "middleware".

Due to the nature of open-source, there were (and still are) several different ways to define middleware. With PSR-15, we now have a semi-official standard, which defines precisely what a middleware should look like. This allows us to benefit from a huge ecosystem of middleware that other people wrote for us. ? Examples might be caching, security and HTTPS redirection.

Next up

I have started and almost completed a rewrite of the most foundational classes that interact when a Flarum application (either the console, or when handling web requests) is booted. This will fix issues such as this one, which have been popping up from time to time. A bit of research and several deep thoughts :thinking_face: showed me that these errors would keep appearing due to some flaws in this initial setup phase. The ground problem was that we were trying to setup the same application instance when Flarum was in different states: not installed and installed (there are other ones, but they are less important). Because some things (such as a database connection) are simply not yet available when Flarum is not installed, we had to jump through hoops to make code work in both scenarios, and were continually creating more such cases when writing new code. The new setup recognizes that these scenarios essentially represent different applications, which are therefore booted differently.

My biggest task for beta 8 is the new PHP API for extensions. This will reduce the amount of boilerplate code needed to write extensions, and also improve the experience of writing extensions, as the new API is much more declarative (and therefore expressive). The groundwork has already been done, the remaining work is a matter of dilligence, as all currently existing use cases have to be made available through so-called "extenders" (check out the issue for some examples and very lengthy discussion). This will likely not be finished completely before the release of beta 8. As the previous method of subscribing listeners to events will continue to work, this is not a problem. In fact, we hope it will make the process of upgrading extensions a little easier, as extension authors can split their work over the remaining beta releases.

Toby

Webpack

This is old news by now, but better late than never: the webpack branch has been merged into master! ? You can read about exactly what this means in one of my previous posts.

All of our core extensions have been upgraded too, and the Travis CI flarum-bot is up and running, churning out new transpiled JavaScript files each time the JS source code is updated. I, for one, welcome our new overlord. ?

JavaScript Extension API

The groundwork for the new JavaScript Extension API is also in place. While we aren't yet actually making use of it in our extensions yet, this will come soon (though it will not block the release of beta 8). It will make use of concepts very similar to the new PHP Extension API that Franz has touched on.

Database Changes

I have also put in some time to finish off what @luceos started with the database naming changes. They are now ready to merge, awaiting review from the others. In the next couple of weeks I will also be aiming to apply similar naming changes to our JSON-API and Eloquent model methods for consistency.

Beta 8 Todo List

Not much left to do now!

Merge database changes (awaiting review)
JSON-API/Eloquent naming changes
@Franz's rewrite of some foundational classes
Review outstanding PRs
Update extension quickstart docs
Write release announcement

Toby

Designing APIs is hard. There are many ways to structure some code so that it "works", but many fewer that satisfy the criteria of being reusable, testable, extensible, and future-proof. With Flarum we're striving for the latter, given our current focus on building a rich, stable Extension API.

Frontend Rearchitecture

With that in mind, I've just completely rearchitected a big chunk of old code that was responsible for rendering Flarum's HTML. From the moment a request is routed to the forum or admin "frontend", the JavaScript and CSS assets are prepared, and then the HTML template is filled out – and everything in-between.

Most notably, the architecture for this stuff now uses composition over inheritance. This makes much easier to build new frontends – in the Embed extension, for example, which will dispaly a discussion without any of the forum chrome so it can be embedded as a blog comments section. It also means that this part of the Extension API is much less likely to require significant changes in the future. Finally, it will make the code a whole lot easier to write tests for.

A collateral benefit is that we now have the ability to fully recompile the CSS when validating the Custom CSS entered in the appearance settings. Previously the CSS was only parsed for syntax errors, but we were not able to validate that it compiles correctly, so invalid LESS variable usage would cause a crash and require a manual database fix.

There are also some further changes to the asset compilation pipeline which will theoretically allow for different filesystem drivers to be swapped in and out, paving the way for use of CDNs.

Up Next

I got a bit sidetracked with this little project, so my todo list is largedly unchaged since my last diary entry. But now it's full steam ahead with polishing off the Webpack changes and getting them into master. I'm hoping to get that done within a day or two.

luceos

YAQU, might as well abbreviate Yet Another Quick Update ?

I've been revisiting the migrations needed for the upcoming beta. The user preferences column is going to be replaced by either a) columns on the users table itself or b) entries on a notification_preferences table. The last causes quite some complexity in code. Some might say that code indentation might be a code smell:

    'up' => function (Builder $builder) {
        // get the eloquent connection from the migration schema builder
        $db = $builder->getConnection();
 
        // start an eloquent builder instance from the users table
        $db->table('users')
            // query only users which have a preferences column with anything set
            // users that never saved their preferences will have a null value
            ->whereNotNull('preferences')
            // order by the user id; which is necessary when chunking
            ->orderBy('id')
            // chunking allows us to loop over the users without breaking down the database
            ->chunk(50, function (Collection $users) use ($db) {
                // loop over all users within this chunk of 50
                $users->each(function ($user) use ($db) {
                    // decode the preferences column (stored as json)
                    // create a laravel collection from it
                    collect(json_decode(Arr::get($user, 'preferences', '{}')))
                        // loop over each preference
                        ->each(function ($value, $key) use ($user, $db) {
                            // save the discloses_online preference into the new users column
                            if ($key === 'discloses_online') {
                                $db->table('users')
                                    ->where('id', $user['id'])
                                    ->update(['discloses_online' => (bool) $value]);
                            }
                            // whenever we encounter a preference related to notify, split it up using a regular expression
                            if (preg_match('/^notify_(?<type>[^_]+)_(?<channel>.*)$/', $key, $matches)) {
                                $db->table('notification_preferences')
                                    ->insert([
                                        'user_id' => $user['id'],
                                        'type' => $matches['type'],
                                        'channel' => $matches['channel'],
                                        'enabled' => (bool) $value
                                    ]);
                            }
                        });
                });
            });
    },

We've decided to split the database changes up, so that they can be reviewed separately. So that's on my todo next. I hope to be able to finish this task as soon as possible, it's one of the largest changes for beta 8 so far.

luceos

I've been pushing forward with the database changes. although not being able to invest as much time as I hoped. Toby has been worked very hard these past weeks to get us forward in different fields, including on my work.

The notification preferences were split up and some minor database mutations in core were added. Some of the core extensions were already made compliant, but it's a tough job if we want to keep backward compatibility.

Please note summers have always been causing slight slow downs in open source. Thanks for your patience and support!

luceos

As I'm on mobile I'm going to post this real quick. Toby has been pushing hard to get to beta 8 these past few weeks. Especially this week has seen a surge of progress.

My task for changing our database scheme was close to completion, leaving out the core extensions for refactor. Toby took it on himself to push forward. At this pace we can expect beta 8 very soon.

Please say thank you for all the hard work so far by liking this post and by supporting us via https://www.patreon.com/flarum or https://opencollective.com/flarum ?

luceos

We should have reported in much sooner than this. But there's a good reason for the silence this past (looks at the last post) month.

Lots of effort has been spent into the last preparations for beta 8. Compared to all previous versions, this is the largest update ever. Here's a little stat count to prove my point ?

beta.1: unknown
beta.2: 127 commits
beta.3: 258 commits
beta.4: 14 commits
beta.5: 265 commits
beta.6: 167 commits
beta.7: 159 commits
beta.7.1: 1 commit
beta8: 586 commits and counting

Commits isn't really saying everything, but it does say something: more contributors and more activity.

Although I have excused myself from intense development to prepare the release of premium extensions via flagrow.io; I am proud to be a part of this driven, motivated and determined team. The renewed focus gained over the past year is really paying off. The new Extenders API by @Franz and the frontend refactoring to webpack by @Toby are amazing and will further lower the barrier to entry for extension developers. Even more so with what @datitisev and @Kyrne have been doing for the revised docs ?

I can't wait for this beta, especially from an extension developer perspective.