atilkan I don't need that much power for the job
I started up a 512MB instance ($5 one) and successfully installed flarum with composer.
luceos Composer has a bad tendency to not work with less than 1GB memory.
Based on my test, php usage was right around 200MB (197MB through the install). On a currently installed system, it rose as high as 256MB. This of course could raise with other variables (possibly more extensions to manage?)
A solution is adding swap and using that. As a alternative to buying more ram, this would probably be the easiest.
atilkan you definitely had errors and it's not because you ran the command wrong or anything - rather I suspect the other programs using the ram. For example, apache2 gobbles ram. Mysql can use quite a lot as well.
If you attempt to run a program that will use 200MB of ram, along a webserver program using 200-400MB plus a database using 75-100MB on a 512MB device, something has to give.
If I was in this situation (and swap was not allowed), I'd likely shutdown the webserver and database and other services using ram temporarily to let the composer update run.
This requires more work and skill with understanding your system. If you want easy and want everything to run smooth and fast, more ram might be your ticket. Otherwise I'd suggest swap ?