In flarum/core2526, we're discussing ways to automatically export everything from extensions. I've been investigating this a bit, and decided to write up an overview to organize my thoughts. This is somewhat of a loosely structured braindump, but might be useful for future reference in understanding how Flarum's frontend comes together. Since forum and admin have the same process, I'll talk about forum.
This post assumes familiarity with JavaScript concepts like IIFEs, ES modules, and scope, as well as some webpack concepts.
Exports, Part 1
We would like to automatically export things from extension and core JS, so they can be used by other extensions. Makes sense, let's go deeper. What does exporting mean?
Beneath all our fancy build tools, at runtime, all we have is a global scope. We can create local/private scopes through closures and IIFEs, and we can structure global scope through nested objects (e.g. flarum.core, flarum.extensions, or even global libraries like $), but at the end of the day, if code A is calling function B, and function B is not in a parent local scope of code A, then function B must be somehow stored somewhere globally.
Flarum's Compilation
Let's think about our current system in this context. When the default webpack config for Flarum is applied via npm run build, a file is created that effectively assigns an IIFE containing the logic and exports of index.js to module.exports. If we were to then use that file directly in the browser, it'd run the logic (e.g. the console.log present when extensions are created via the extension generator), and provide the exports under a variable (if imported via import in a module script block).
But... that's not quite what we want? Extensions are meaningless without core. So, how does that all come together?
Our compilation process essentially concatenates core code and all extension code. So our compiled file will look something like:
// A bunch of core code
// Code for extension 1
// Code for extension 2
// Code for extension 3
But just recently, we said that compilation creates an IIFE that runs top-level logic and sets module.exports to the exports of index.js. If we just blindly concatenate everything, the logic part would run, but we'd just end up overriding module.exports 50 times. Oh, and for that matter, module.exports wouldn't really accomplish our goal (making data and functions accessible from elsewhere), since we'd be loading the file in a <script src=""></script> HTML element. That module.exports can't be used.
So what do we do instead?
Alright, so now the exports from the index.js files of both core and extensions are globally available to import! And when the browser loads single compiled file, it'll run all the IIFEs (to get the output), and in the process, run any top-level logic. So if you put console.log('hi') as the first line of the index.js of your extension, it'll run when the page loads.
Before we continue thinking about exports, let's briefly cover how Flarum's frontend boots.
Flarum Frontend Boot
Although they sometimes feel like it, browsers aren't magic. The essentials of loading a webpage are relatively simple: the browser gets an HTML page from the server, and goes through it, rendering CSS/HTML and running JS as it goes.
Flarum is no exception. However, as an SPA, it's a little more interesting. The HTML originally returned by the server is super simple (it's what you see when there's a frontend error, or if you visit a Flarum site with javascript disabled). However, we also send a bunch of JS code that essentially rewrites that base HTML to make Flarum the beautiful, interactive software it is. This is why there's no HTML theme templates to edit: the interface you see is created dynamically by JS.
Let's go through the relevant bit of Flarum's default HTML. You can follow this yourself by going to view-source:https://discuss.flarum.org (or if that doesn't work in your browser, just viewing the source code of this page).
// Head metadata and stuff
// CSS files
// HTML for the simple site skeleton
<script>
document.getElementById('flarum-loading').style.display = 'block';
var flarum = {extensions: {}};
</script>
<script src="https://discuss.flarum.org/assets/forum-bd715a46.js"></script>
<script src="https://discuss.flarum.org/assets/forum-en-316c0d1d.js"></script>
<script>
document.getElementById('flarum-loading').style.display = 'none';
try {
flarum.core.app.load({PAYLOAD DATA, THERE'S A LOT OF THIS});
flarum.core.app.bootExtensions(flarum.extensions);
flarum.core.app.boot();
} catch (e) {
var error = document.getElementById('flarum-loading-error');
error.innerHTML += document.getElementById('flarum-content').textContent;
error.style.display = 'block';
throw e;
}
</script>
// Some cloudflare stuff
So, what's going on here?
- We make the loading spinner element visible while the page is rendering
- We create that global
flarum object we talked about in the section above (remember, core's exports are assigned to flarum.core, and all extension exports are collected into flarum.extensions).
- We load the compiled application JS via a script tag. Remember that this runs top-level logic, and collects exports into the
flarum object we just created. Part of core's top level logic creates the global app object.
- We load a JS file containing translations and (if applicable) localized dayjs config.
- We run
flarum.core.app.load on a huge object of payload data. This data is immediately available to extension JS without needing to make additional API requests. It includes some settings, the initial locale, the current user, some preloaded data (for example, the discussions on the homepage), etc.
- We run
flarum.core.app.bootExtensions and flarum.core.app.boot. This actually loads up the application: after this completes, the forum will be usable and feel like a normal Flarum forum.
Note that in steps 5 and 6, we're using the global flarum object that was created when the browser loaded the big compiled JS file.
You might be wondering: "If extension top-level logic runs in step 4, but the app isn't booted until step 5, how can extensions have an effect?" If you look closely at extensions, the only top-level logic you'll see in most is a call to app.initializers.add(). The actual extension logic is in that callback. Flarum runs all these "initializers" during boot in step 5.
Back to Exports
Alright, detour over. We know that code that should be accessible to extensions can be exported by putting it in some global variable (some sub-object of flarum). Currently, we can do this in a clean way by exporting the logic that needs to be exported via ESM export statements in the extension's index.js. Webpack and the Flarum JS compiler will then put that in flarum.extensions for you. But here we face an issue.
If we were programming for NodeJS, we wouldn't need to assemble everything via webpack at all: we could just import the code we need from each module directly. But we don't want to load thousands of separate JS files every time we load a website: that would be chaos! Therefore, we use webpack to combine all the code for core and each extension into one file each. Each of these files entails a module. But if the files are generated from each extension's index.js, how do we export stuff that's NOT in index.js? Most extensions will have at least a few files, and we need to export classes and functions defined in those too.
Right now, there are essentially 1 approach way to do this: exporting an object with all the exports from index.js. There's several variations:
- In core, we create a giant object called "compat" and export that. This compat object contains EVERY class, function, variable, etc that's exported as a default export from some file. So if we have a line
'utils/PostControls': PostControls, we're saying: "the default export for flarum/forum/utils/PostControls is this PostControls class". We could also have named exports by exporting an object with keys being names ('utils/someObj': {a: 1, b: 2}). We can't replicate both at once though.
- Some extensions add stuff to the global
flarum.core.compat object. For instance, tags
- Some extensions export a stuff directly from
index.js. For instance, my Rich Text extension. At first glance, this looks like we've managed to sneak multiple modules into one file, but that's just a facade: the only difference between this and (1) is that in (1), we are exporting a single-level array, and in (3), we're exporting nested objects.
How about imports?
Now that we understand how the exports system currently works, let's quickly go over its counterpart: importing! For imports from local libraries, webpack will automatically bundle that library in the extension's compiled output. But how do we import stuff from core, or from extensions?
As you might have guessed, the answer comes back to that global flarum object. The Flarum webpack config instructs webpack to retrieve Flarum core and extension imports from that object.
JS Extenders, and why "compat"?
The backend extension API is very use-case-driven: there are certain extenders, which present the public API, and are used by extensions. Read the docs for more on this.
The frontend extension API is very different: everything is based around modifying global state and monkey-patching on logic. This offers a great deal more flexibility, at the cost of some stability and higher potential for conflicts.
Originally, there were plans to make the frontend API much more similar to the backend one. However, this never fully manifested, and as we approach stable, it'll have to be delegated to future major releases (Flarum 2.0?). The current system is named compat because it was intended to be a compatibility layer before moving to the new API: see this GitHub comment for more information
In the long term, I think investing in extenders for Flarum makes sense. We kind of already have them for registering settings/permissions in admin! But I don't think we'll ever want to close off the monkey-patching modifications; the current frontend system allows nearly any part of the UI to be changed, and I think we want to keep that. Instead, sometime after 1.0, we'll probably introduce extenders for some things:
- Registering models
- Registering routes
- Registering admin settings/permissions
- Translation pre-processors
And other stuff, mostly to do with the global app object / application state. I don't think we'll want to do so for components, although that approach might change with time.
So Now What?
We want to automate exporting code defined by extensions. Luckily, since we have control over the webpack config, we have a layers around both importing and compilation where we can insert custom logic.
As I've covered in this post, any solution will need to somehow put stuff into a global object. Even if it's not called flarum.extensions, the concepts (and limitations) are the same. There's no magic.
There are several approaches we could take:
- We could add a command to Flarum CLI that would generate a
compat.js file. That requires extra code though, and it's easy to forget to run this.
- We could try and intercept module construction to add additional exports. This would be extremely complicated (we'd be hacking webpack internals, and that stuff is hard) and difficult to maintain.
- We could use a variant of exports loader or modify source plugin to append all the exports we need to
index.js during the build process, without actually modifying the file in source. I think this is the best option: extensions could pass a "include" and "exclude" params to the webpack config, specifying which files should have their imports included or excluded. Then all we'd need is to use fs to go through the extension's code at compile time, figure out what files are exporting what, and put that into a standard format at the end of index.js
Personally, I think the 3rd option is best. Although any mechanism works as long as extension developers are able to provide "include" and "exclude" lists, and if the final result ends up in flarum.extensions.
While we're at it, we should probably also try to support default imports at the same time as named ones (if a default key is there, use that). And we should handle nonexistent imports from extensions more gracefully. Luckily we have full control over importing through webpack config.
