Flarum CLI Dev Diary

askvortsov

In https://discuss.flarum.org/d/26525-rfc-flarum-cli-alpha, we announced the alpha stage of Flarum CLI. We envision this as a CLI tool that will generate boilerplate, update extensions, perform auditing/validation, and all sorts of other cool stuff that makes development easier. So far, we've effectively built out a "proof of concept" for the major categories of functionality:

boilerplate creation
code generation
updating for changes in Flarum
updating extension infrastructure

(sorry audit, you're just too complicated for now. Someday 😥)

Now that this is working, we need to start expanding functionality, especially when it comes to code generation. We're planning to support generation of everything from JS components to backend models to serializers, extenders, migrations, test cases, and practically everything else you might need in Flarum.

Problem is, this gets complicated fast. Just look at the current code for generating event listeners. It's a confusing, jumbled mess. If we copy-paste this for all the different stuff we want to support generating, it'll be an absolute pain to maintain. And what if we want to support more complex scenarios?

Adding extenders to extend.php, but not boilerplate stub files
It'd be cool if, when generating a model, you'd also get the option to generate API controllers for that model. And a serializer. And maybe a validator too? If we just code that up naively, that's going to be a thousand-line-long file. No one will EVER want to touch it.

So in this discussion, I'll give an overview of how the Flarum CLI is currently designed, and how/why we might want to implement code generation.

Current Design

Background

Flarum CLI is the planned successor to the FoF Extension Generator. The old generator was relatively simple: it prompted the user for various config, copied over an extension skeleton from a template, filled in variables from the provided config, and wrote the new files into your filesystem. Hooray, new extension!

CLI Design Overview

The CLI itself is built around the oclif framework. This takes away all the boilerplate work of making the CLI, well, a CLI. Subcommands are nested in folders, and each command corresponds to a file, which exports a class extending the oclif-provided Command class. Simple enough.

Anyways, back to structure. Most commands works in the following process:

Confirm that we're in a Flarum extension (unless we're generating a new extension), and get the current directory.
Prompt the user for whatever config data we need
Create or modify files to accomplish the command's goal using the provided config. This is done in an in-memory filesystem so that if something breaks, we don't create a bunch of unneeded stuff.
Once everything is done, commit changes to the filesystem. Congrats, you now have a new... something!

File Creation and the PHP Subsystem

Let's explore step (3) a bit deeper. There's currently 3 general "types" of file changes/modifications we do.

Read files, look for simple patterns via regex, modify them, and write back to the file. This is what we do for the update js-imports command.
Copy in a boilerplate file, replace variables with values provided via config. This is what we do when we generate migrations, or initialize a new extension.
Modify an existing file to add/change non-trivial code. This is what we do when automatically adding extenders to extend.php.

1 and 2 need no explanation, but 3 sounds pretty complex. Well, luckily for us, there's an awesome library that allows parsing PHP code into an AST, making modifications, and turning it back into PHP, with relatively minimal changes to code style. Yay! Unluckily, it's in PHP itself, so our use of it will also need to be in PHP. But the CLI itself is in JS/TS. We solve this by including a mini PHP package as a subdirectory of the CLI source code, and calling that from JS via the child_process node library. Essentially, we call the PHP code from our JS code.

Improvement Proposal

Currently, business logic of commands (ie the part where files actually get moved around and stuff) is located directly in the commands. We try to DRY some stuff by using a custom subclass for commands, but that's a weak solutionThis is great for the alpha version, as we just write the code we need, but as I explained in the beginning, this doesn't scale well. For that reason, I think we need to separate business logic out into a new layer. For now, this layer should focus on modification types (2) and (3).

I think a class with a fluent API would be a good candidate here:

(new GenerationTool(rootDir, currentDir)
    .copyBoilerplateDirectory("extension/*", "$CWD")
    .copyBoilerplateFile("extension/.github/workflows/test.yml", "$EXT/.github/workflows/test.yml")
    .copyStub(Generation.Migration, "$EXT/migrations")
    .copyStub(Generation.Backend.EventListener)
    .addExtender(Extenders.Event.Listen)
    .updateComposerJsonFieldFromInfra('scripts.*')
    .updatePackageJsonFieldFromInfra('scripts.format')
    .promptForMissingConfig(function (missingConfig) {
        return prompts(missingConfig);
     })
    .execute()

$CWD is the current directory, $EXT is the extension root. We should also have a magic $BEST_DIR, which would be either the user-provided directory (if explicitly provided) or a "best practices" directory as defined in the schema (see below). For example, if you're generating an event listener and don't say where to put it, we should put it in the src/Listener directory. This will make development easier for new devs by eliminating decision making about folder and file structure.

Generation and Extenders could be some form of enums, acting as identifiers for a "Schema" system. This would allow defining "schemas" for extenders and stubs. The purpose would be two-fold:

The schemas for extenders would be populated with user-provided config, and turned into Extender Specs, which is how we tell the PHP system what kinds of extenders to add. This way, we could define a single schema for use in potentially many different generation commands, instead of manually assembling the extension spec directly in the command logic ewwwwww....
We could figure out which user-provided config is needed, what types/validation it should be, and generate a list of missing config.

The part I'm least sure about is providing config values. Should we do it in each step, or all at once at the end? Or both? All at once at the end has the benefit that we can provide some values, and the generation tool will check which ones are missing. Then we can prompt for those. On the other hand, we'd need to namespace them to prevent conflicts. I think the solution here will become apparent as we try out various approaches. This is all private API for internal consumption so BC is not a concern.

askvortsov

System Architecture

askvortsov Anyways, back to structure. Most commands works in the following process:

Confirm that we're in a Flarum extension (unless we're generating a new extension), and get the current directory.
Prompt the user for whatever config data we need
Create or modify files to accomplish the command's goal using the provided config. This is done in an in-memory filesystem so that if something breaks, we don't create a bunch of unneeded stuff.
Once everything is done, commit changes to the filesystem. Congrats, you now have a new... something!

Looking over this again, I think actual behavior is a bit more complex. Here's what my initial post describes:

But that's not entirely accurate. For example:

Generating new boilerplate will do this. But it also has additional steps where it asks the user whether composer install and npm install should be run.
My idealized example of prompting to generate controllers, migrations, etc after creating a model also consists of a chain of steps.

So instead, let's do:

This is already a lot cleaner. These steps can now be reused across multiple commands. We've successfully decoupled implementation logic from the interface of running commands, yay!

Now, we need to define our steps. Ideally, we want our steps to be as granular as possible so we can use them in all sorts of different places. So in creating a new extension, the steps would be:

Initialize skeleton from boilerplate
Run npm install
Run composer install

This works. How about if we want to generate a new event listener?

Create an event listener file
Insert an extender into extend.php

It still works, but not quite as elegantly as with the previous example. There's several issues:

This setup makes it possible for a listener file to be created, but not inserted into extend.php
There's a significant overlap of settings, and users will need to enter them twice.

To solve this, let's consider the following "mathematical" definition of a step:

A step is a function that takes (1) the current files in an extension and (2) a set of user-provided parameters, and outputs (1) new files for the extension and (2) a new set of parameters.

This interpretation is extremely powerful, because it allows for composition of steps. In other words, we can define our steps as granular, atomic processes. But then, when we actually use the steps in commands, we can "chain" them together (meaning that params are shared, and file changes are commited together when all chained steps succeed) instead of just executing sequentially.

This approach also simplifies the steps themselves by moving "do we want to do this step?" and "commit changes" out of the step definition and into a "step manager". Presumably, the former logic is provided by the command, and the latter logic is the same for all steps (since it can be applied to any step). Let's visualize this new design:

Unfortunately, not all steps will fit under this nice mathematical model. For example, the composer install step will directly modify the filesystem, meaning its "do something" and "commit changes" steps are the same step. Since it's no longer a "semi-pure" function, we can't compose it with other steps. In the future, we could investigate ways to make it composable, but for now, we'll just throw in support for non-composable steps. Let's update our flowchart one more time:

A few other considerations:

Steps should not be aware of how parameters are obtained. They should declare the parameters they need, and the step manager should be responsible for getting those parameters and providing them to steps.

luceos

Would a statemachine be a solution here?

You define the pipeline per command, eg:

remove a file
ask for user input
write a file
commit

A statemachine can store user input between steps and understand when more information is required as it can conditionally dispatch the next step based on requirements. This would also allow skipping some stages when files already exist or dispatch alternate steps based on user input.

askvortsov

luceos You define the pipeline per command

To be competely honest, that's the part I'm most afraid of. We're going to have quite a few of these commands (at least 20, probably more), and I would like it to be as simple as possible to add new ones and maintain existing ones. By moving the implementation logic into steps, we decouple the actual implementations of these steps from the CLI / user input layer. Furthermore, since the vast majority of steps are implemented as simple, "semi-pure" (they mutate the input virtual filesystem), granular functions, we can easily add a unit test per step to confirm that the proper files are affected. Note that since composable steps don't actually edit the filesystem, we can do these tests in complete isolation from the environment.

Back to maintainability, I fear that making each command a state machine would result in a lot of duplicate boilerplate for updating and managing this state.

askvortsov

Started implementation on this. It's looking a lot better than the previous "schema" idea, but there's still a few shortcomings:

Composed vs Atomic

I'm not sure "composed steps" are actually useful the way I intended them. Instead, we're going to aim for "atomic groups" of steps, where if something goes wrong in one of the steps (e.g. there's an error, user exits, etc), none of the atomic steps will be applied. That being said, steps in atomic groups can still be optional.

True step composition (combining multiple steps into one true step) is still an interesting concept, but since atomic groups are more useful for what we have planned, we'll go with those for now.

Dependencies and Sharing Parameters

We're still lacking a solid way to pass parameters from one step to another. For example, one step could create a model instance, and the second could create a controller for that model. We'd want to pass the name of the created model class from step A to step B.

We're also missing support for step dependencies: for example, step D should run only if step B runs.

After some thought, I believe that these 2 concepts are equivalent.

Assume step B relies on parameters from step A. Therefore, step B is dependent on step A.

Assume step B is dependent on step A. Each step contains a relatively granular set of file operations. So a dependency implies that step B does something with / based on the files modified by step A. But then, it needs to know something about what step A did. Therefore, there needs to be some flow of information from step A to step B.

We could accomplish this by adding 3 things:

When registering steps, we could provide ids.
When registering steps, we can provide an list of arguments it depends on. Each argument would contain the ID of the step that provides it, the name of the param in the step that provides it, and what it should be passed to the consumer step as.
On each step, we can add a method that declares which params it provides.

StepCollection usage example

const steps = (new StepCollection())
    .step(new UpdateJsImports())
    .step(new BackendTestingInfra())
    .namedStep('model', new GenerateModel())
    .step(new GenerateShowApiController(), false, [{
        sourceStep: 'model',
        exposedName: 'modelClass',
    }])
    .step(new GenerateApiSerializer(), false, [{
        sourceStep: 'model',
        exposedName: 'modelClass',
        consumedName: 'targetModelClass', 
    }])
    .atomicGroup((stepCollection: StepCollection) => {
        stepCollection
            .namedStep('listener', new GenerateEventListener())
            .step(new EventListenerExtender(), [
                {
                    sourceStep: 'listener',
                    exposedName: 'listenerClass',
                },
                {
                    sourceStep: 'model',
                    exposedName: 'modelClass',
                    consumedName: 'isnt_used_here_but_why_not', 
                }
            ])
    })

stepManager.run(steps);

Desired behaviors of StepCollection

Can add complex but valid sequence of steps (e.g. what I have above) without issue
Dependencies on non-existent steps will error when being added.
Dependencies on non-existent exposed params from existing steps will error when being added
Named steps must be unique

Desired behaviors of StepManager

Can run a sensible sequence of steps
If a step doesnt run, its dependencies won't be run.
Any step can be optional. If it's optional, and the user doesn't confirm it, it won't run
Params are properly passed to dependencies.
Changes made by steps in atomic groups aren't committed till the last step in the group executes successfully.