[RFC] Drop Composer, simplify project structure

Toby

Context: https://gitter.im/flarum/internals?at=583d3b88381827c24d945f22

Problems with Composer

Composer uses a lot of memory, probably too much to run reliably on shared hosts. After doing some investigation, I don't believe this is something that can realistically be fixed. (A very in-depth understanding of the Composer codebase would be required, and even then it seems that high memory usage is just inherent in the way Composer was designed.)
- We are not using Composer the way it was intended to be used: "you are not supposed to run composer update on your production server. You should run update on your dev box then deploy the composer.lock file and run install which will run with very low memory footprint and very fast since it doesn't have to check for new packages but merely applies what's in the lock file." source
- Workarounds include setting up a local dev environment (provide a tutorial) so people can run composer locally (with their own selection of extensions etc.) and then upload the resulting vendor folder. Or even an app, or a service that does it for you. Theoretically this could run seamlessly in the admin interface - you click "update" and it sends your composer.json file to the service, then downloads a zip and extracts the new vendor files.
- However, requiring a service to simply update an app and install new extensions seems needlessly complex and expensive.
We have no way to "lock" third-party dependencies. Since each individual Flarum installation uses Composer to download third-party deps, we can't guarantee what version they will get (beyond a version constraint). Just recently, this caused a problem with zend-stratigility and the deprecation notice. We can't solve this with a composer.lock file because every installation will have different extensions. I'm not sure how much of a problem this will actually be in the real world, but theoretically when debugging a problem we can't just ask someone "what version of Flarum are you using" – we need to ask "what versions of each individual package are you using"!
We've only actually solved half of the dependency conflict "problem". Part of the original motivation to move everything to Composer was to avoid dependency conflicts been extensions. (i.e. if two extensions included/autoloaded two different versions of the same third-party lib, we might get hard-to-debug errors.) However, as pointed out by sijad, this problem still exists with Bower dependencies. For this there's no feasible solution – it's just something we have to deal with.
We have to commit our dist files to our repo because Composer only downloads packages from GitHub. This bloats the repository size and makes PRs more prone to conflict and to contain irrelevant changes (eg. if developers are running different versions of Babel).

The alternative

Maybe we should consider reverting back to how things used to be (pre-beta 3), with a few simplifications. This is also more or less how Drupal, phpBB, and MediaWiki are set up:

Just one repo (flarum/flarum). Get rid of the flarum/core repo – move the core code into flarum/flarum. This greatly simplifies our presence on GitHub. Requiring flarum/core directly into a Laravel project was never really going to work; integration will be much more elegantly performed by interacting with the JSON-API of a standalone Flarum installation. If someone really wants to, there's nothing stopping them from requiring flarum/flarum in the Laravel project – they'll still get access to the core classes!
Consider moving the core extensions into flarum/flarum too. If we want upgrade to be easy, people should just be able to download the new version and replace their core directory. They shouldn't need to download new versions of each basic extension and replace it in their extensions directory. Rather, the core extensions should be contained within a separate core/extensions directory – they are still "extensions" and they can be disabled, but they cannot be uninstalled in the same way as other extensions. Again, this is how Drupal 8 does things with its core modules. Also, this would obviously simplify things like issues on GitHub.

Caveats

The possibility of third-party dependency conflicts between extensions. But how much of an issue will this actually be? How often will extensions actually include third-party deps, let alone the same one, let alone incompatible versions of the same one? The auth ones don't need to anymore as oauth2-client is included in core. Other PHP software (WP, Drupal, etc.) seem to deal with it fine. It's an issue we'll have to deal with for bower deps whether we like it or not – maybe we should just deal with it for PHP deps too?
We will have to build our own little extension package management system, but that shouldn't be too hard and it can come at a later date (for now we can just support installation and upgrade via FTP).

Dominion

Toby Consider moving the core extensions into flarum/flarum too.

It seems to me this would make extension updates dependent on the core update cycle, and vice versa. So if a small but critical bug is found in just one extension, users will have to replace everything, rather than just update that one extension ... which seems like a bit of unnecessary fuss. You'd be forced to test all the extensions thoroughly before releasing a core update, to avoid that sort of thing. Which could slow down the release cycle as a whole, no?

Toby

Dominion It seems to me this would make extension updates dependent on the core update cycle, and vice versa.

Yes and no. Philosophically, we would treat our core extensions just like any other software treats its core features, i.e. they are not "extensions", they are just features that a majority of people will use but that can be turned off for the minority. If you think about it like this, then I don't think the coupling is really an issue.

Practically, once we get to a stable version, we won't hold back with the bugfix releases (1.0.x). If we need to distribute a new point release to fix a critical bug in even a small less-used feature, that's fine. If only a few files are changed, then we can instruct people that they only need to replace those files if performing a manual upgrade.

Dominion

Toby If only a few files are changed, then we can instruct people that they only need to replace those files if performing a manual upgrade.

Okay, that sounds reasonable. Perhaps we can even make a delta package available with just those files and instructions on how to replace them.

jordanjay29

There's another philosophical issue I find. Once we merge the extensions into core (or flarum), we'll lose the moral high ground when faced with complaints that "everything is an extension." We've thus far been able to point out that even Flarum's core features are extensions that they can remove completely if they want, or update independently, fork, etc. Merging it in, while it doesn't lose this ability completely, does make it a more difficult argument to make.

Honestly, I like the setup as it is now. It makes it easier for someone to create a drop-in replacement for, say, the Approval extension and with two composer lines replace the Flarum version. I'm not opposed to this entirely, I just prefer the current setup.

Merging core into flarum makes more sense, though.

Dominion

jordanjay29 We've thus far been able to point out that even Flarum's core features are extensions that they can remove completely if they want, or update independently, fork, etc. Merging it in, while it doesn't lose this ability completely, does make it a more difficult argument to make.

I agree, it would be a slight departure from how the project has been described to date ... even if it is only a matter of appearances, it would (at the very least) have to be explained carefully.

I think it makes great sense to include certain extensions in the core package. This is especially true of the default English language pack, since Flarum will have a hard time displaying anything without a language pack installed. But many extensions could easily be treated as optional installs.

jordanjay29 Honestly, I like the setup as it is now. It makes it easier for someone to create a drop-in replacement for, say, the Approval extension and with two composer lines replace the Flarum version.

It might be possible to achieve something like the current setup (extensions optional) without Composer by including only the core in the downloadable installer and having the setup routine download and install the extensions. The installer could have a checklist for optional extensions, so the user could avoid downloading any extensions that are not needed.

But this would mean prioritizing the extension package manager that @Toby mentioned at the end of the OP, as it would be necessary to handle extension upgrades.

Toby

Dominion ... even if it is only a matter of appearances, it would (at the very least) have to be explained carefully.

We would definitely need to explain it carefully... but as long as we did, I don't think we'd lose the "moral high ground". The philosophy is still there – the code is still just as modular and just as optional. It's merely a logistical change to make everyone's life a bit easier.

Dominion It might be possible to achieve something like the current setup (extensions optional) without Composer by including only the core in the downloadable installer and having the setup routine download and install the extensions.

Exactly, this might be a long-term goal, if there is truly benefit in having our core extensions as separate packages... but as you said, until we get a proper extension package manager together, having separate packages would certainly make upgrades very tedious. Since our priority is to get a stable version out, maybe we should include them in core for the time being, and consider extracting them later.

(I still think it is ultimately more manageable for the project to have a single release cycle for all of the features being worked on by core developers, rather than individual release cycles for each extension.)

jordanjay29 Honestly, I like the setup as it is now. It makes it easier for someone to create a drop-in replacement for, say, the Approval extension and with two composer lines replace the Flarum version. I'm not opposed to this entirely, I just prefer the current setup.

The equivalent with the new setup would be: disable the Approval extension, copy core/extensions/approval into extensions/my-approval (for example), make changes and enable. Arguably even easier ?

jordanjay29

Toby You can always make changes to your own Flarum easily. I mean for someone who wants to make their changes distributable. ?

Toby

jordanjay29 Right, so couldn't they then just zip up extensions/my-approval and distribute it? I don't see the benefit of Composer you're advocating.

Dominion

Toby Exactly, this might be a long-term goal, if there is truly benefit in having our core extensions as separate packages... but as you said, until we get a proper extension package manager together, having separate packages would certainly make upgrades very tedious. Since our priority is to get a stable version out, maybe we should include them in core for the time being, and consider extracting them later.

That sounds like a reasonable plan.

Toby Right, so couldn't they then just zip up extensions/my-approval and distribute it?

Basically, back where we started: unzip, upload to extensions, and enable. I get the feeling that this will please a lot of people who were bothered by the whole idea of Composer in the first place.

... In fact, I'm somewhat dreading the flood of told-you-sos we're sure to see in the forums ... ?

Dominion

Toby I just noticed ... that image you posted has separate folders for extensions, languages, and themes.

Is that what you're thinking of doing for Flarum ... separate folders for different extension types?

Dominion

After trying to help a translator figure out where he went wrong I'm starting to feel more and more like dropping Composer might be the best route. I think we had more people testing Flarum (and giving better feedback) back in the pre-Composer days. Since Composer, I sense we're spending proportionally more time with troubleshooting of installation issues and problems such as the one I linked above ...

Dominion Is that what you're thinking of doing for Flarum ... separate folders for different extension types?

... If so, I think that's a good idea. If possible, I think we ought to aim for a system whereby admins can just upload the ZIP for each new extension to an "uploads" folder, then click an Install button in the admin UI to unpack the uploaded file into the proper directory. That sort of system seems relatively foolproof and it's easy to explain.

Davis

Dominion admins can just upload the ZIP for each new extension to an "uploads" folder

Better yet, link a GitHub repo then select the release?

Dominion

Davis Better yet, link a GitHub repo then select the release?

Sure! ... Though I think that if we're going to be moving away from Composer, we ought to take advantage of the freedom that creates and allow folks to deliver their extensions in whatever format they're comfortable with.

GitHub's a no-brainer for devs, but (as I've pointed out to Toby) it may be new to many localizers ... who may be happier with the idea of just editing the text files and zipping the whole thing up for delivery.

Davis

Dominion who may be happier with the idea of just editing the text files and zipping the whole thing up for delivery

I feel like there's need to be more organization than that.

I brought up the idea with GitHub because php can easily manipulate GitHub API to fetch packages without admins downloading anything.

Dominion

Davis I feel like there's need to be more organization than that.

When it comes time to set up the Flarum Marketplace, you're probably right. But requiring GitHub as part of the localization workflow is bound to cut down on the pool of translators who want to have anything to do with Flarum.

luceos

One of the benefits so far of Flarum has been the level of quality provided by its authors ánd her extension developers. Composer has always generated a certain level of entry to overcome, which - at the same time - also prevented less experienced developers or developers less willing to adhere to conventions to contribute to Flarum.

The risk I foresee by allowing arbitrary zip file uploads is moving closer to an ecosystem Wordpress has proven is not safe. A lot of our users could be classified to fall in the same group as a Wordpress user and as such will most likely make the same mistake.

The only way I know such a system might work is by integrating some kind of version/sanity verification of plugins, natively into Flarum. Aside from doing version checks on installed extensions, doing checksum verification on files will help in protecting our users from ill intent. Or you could implement a market place which, aside from these safety guards, would also be able to allow for community driven QA.

Now whether you want to take these things into considerations at this point is up to you, but .. not taking into consideration any of this will eventually affect the great image in the community Flarum currently has in relation to stability and security.

luceos

Having thought about this some more and having looked at this issue about extension php module dependencies I am even more unsure that zip installation will be a solution for Flarum. The increased adoption from dropping composer might be less valuable than the increased workload for all issues related to dependency conflicts and missing dependencies (no curl installed f.i.).

Having said that, we could do a few things ourselves, if we still continue to use composer underwater. For instance extensions won't commit the vendor folder and Flarum will include the composer binaries to install dependencies by itself or report issues back. The process would then look like the following:

upload zip or files (let's go for the latter now, the former is just an additional step)
show the extension name in the extensions page
once enabling check dependencies, try including the package (repository path or something) and then running install for that specific package
read the response and provide feedback to the user

What do you think?

Toby

luceos The risk I foresee by allowing arbitrary zip file uploads is moving closer to an ecosystem Wordpress has proven is not safe.

Absolutely, we will take the safeguards you've suggested into consideration in the future when we set up our marketplace.

luceos The increased adoption from dropping composer might be less valuable than the increased workload for all issues related to dependency conflicts and missing dependencies (no curl installed f.i.).

The motivation for this change is not only to increase adoption. There are also other issues as I've outlined in my first post. I think Composer is really the wrong tool for this job.

@Franz if you have some time it'd be great if you could read through this discussion and provide your own thoughts. I'm happy to continue debating, but we need to reach a conclusion soon so we can move forward.