Cloudflare issues

MikeJones

jordanjay29 would love to learn from the teams experience with this attack.

I know everyone is busy but at some point would love to read an after action of what went down. What were the warnings signs etc. What files and logs did you check, what did you guys see etc.

Super interesting and would be a great learning experience for all of us hosting our own sites.

Toby

MikeJones I woke up the other morning and couldn't access the forum. DigitalOcean monitoring showed CPU usage at 100% for the past couple hours, with the nginx process accounting for most of it. I ran tail -f access.log to see that there were dozens of empty requests from a range of IPs hitting the server every second. I proceeded to set up CloudFlare and switch over the flarum.org nameservers, and created a new droplet with a different IP.

MikeJones

Toby awesome thanks for the info!

Nathan5226

Courageous!

010101

Hmm. Ok, maybe I should set up Cloudflare after all.

RelatedTitle

I don't know about this but I've recently been seeing a couple of spam posts leading to some diet tip or something website, maybe this could be the same people attacking it, perhaps it's listed on a vulnerable site lists, I see a lot of those in pastebin both could have come from the same source.

010101 It's free but goes really bad with flarum, you will have a bunch of cache-related issues.

FlarumPower

Begs to ask the question , why would anyone want to (D)DOS an open source project website?

Justice

FlarumPower Just the question I was asking myself

010101

I don’t know much about hacking or DOS attacks. But, maybe they just pick a bunch of random sites to attack and this happened to be one of them. Some people just want to watch the world burn. They don’t have good reasons. Or, someone here hurt their feelings.

Who did it? Who hurt someone’s feelings recently?

clarkwinkelmann

RelatedTitle you will have a bunch of cache-related issues

it depends how often you install/update extensions or customize your forum theme. As long as you clear everything on your server and on Cloudflare each time you shouldn't run into any more issue than without Cloudflare.

I personally prefer to clear Cloudflare cache manually when needed but if you don't feel like doing that you can use the Uncache extension by ReFlar to do it for you automatically ?

In beta8 with the assets versioning/naming fix (I believe it's fixed) you probably won't even have to touch Cloudflare at all anymore as different versions of assets will always have a different name so won't be incorrectly cached.

luceos

As there's nothing to gain from keeping silent about the recent issues, let's give an update.

So .. we assumed the DOS would pass over; however the attack is now (possibly) periodically returning as a D(istributed) DOS; meaning the origin of the attack are multiple ip addresses/servers. Last night starting 22:00 an attack surpassing 20 million request was targeting discuss.flarum.org. Luckily around that time both me and @jordanjay29 had been looking at the forums and the Cloudflare increased protected was enabled ("I am under attack"). The benefit of this mode is that all clients have to wait and be scanned before allowing access. The drawback is that now and again your session would get blocked and you'd get that nice red alert banner on the forum telling you there was an error.

As I was already in the process of setting up monitoring for my own projects with prometheus and grafana, I proposed to include Flarum too. We could easily identify the higher load on the droplet before the increased CF protection.

I decided to improve our fail2ban settings, first by blocking failing ssh requests sooner and then with some additional protection via nginx using the http req limit module combined with a custom filter for fail2ban. While temporarily reducing the CF protection one specific IP from Mexico was soon banned as a result. The droplet still couldn't cope as the PHP FPM pool was configured to allow a maximum of 30 (which was simply too much) FPM processes to be started. Please note discuss.flarum.org is running on a 1 CPU/2GB RAM droplet which is outright sufficient under normal circumstances.

We'll be improving our line of defence in the coming days.

MikeJones

luceos thanks for the update! Always looking to learn and see how I can better prepare my flarum from attacks.

luceos

We're under attack right now. We're not increasing the CF protection for a bit to see how well the droplet now is able to handle the attack by itself.

Seems like we are holding off, cpu is still in use; but nothing we are not able to manage.

I've implemented a little something that will ward of new attacks faster ?