Optimisation of Robots.txt file 2025?

Wellwisher

Optimisation of Robots.txt file 2025 / Downtimes

I run a small celeb forum. Today, my forum came to a halt, causing downtime.
My VPS provider said this was because of the website being hit heavily by "Bot Crawlers":

Flarum information (Response from my VPS provider):

Hi, after checking, it appears like the website is being hit by bots heavily,
which has caused an overload that resulted in the downtime, here's a list
of the hits:

92 PetalBot
102 Pinterestbot
126 GPTBot
151 YandexBot
159 Bytespider
325 SemrushBot
802 DotBot
1189 AhrefsBot
2118 bingbot
3054 Googlebot

Numbers on the left e.g "3054" Googlebot represent the number of requests being made (I am assuming).

Now the problem is, I want my site to be known. but not at the expense of interrupting my community and it's members.

The bots don't need to crawl EVERYTHING.

- Is there an Optimised Robots.txt version that Flarum uses? So I can command bots not to crawl unnecessary things like "users" for example?

- How are other webmasters tackling this issue?

RelicSystem

I would spend the time figuring out which ones are hurting my server and then outright get rid of them, others if they support it introduce Crawl-delay to slow them down a little. You'll find dozens (if not more) projects on GitHub with the 'ultimate robots.txt files' but they all have their own goals so take what you want.. scrap the rest. (example: mitchellkrogza/apache-ultimate-bad-bot-blockerblob/master/robots.txt/robots.txt)

Wellwisher

I am experimenting with crawler delays.

Step 1

# Allow these bots with a 10-second crawl delay
User-agent: Pinterestbot
Crawl-delay: 10

User-agent: GPTBot
Crawl-delay: 10

User-agent: YandexBot
Crawl-delay: 10

User-agent: bingbot
Crawl-delay: 10

User-agent: Googlebot
Crawl-delay: 10

# Disallow all other bots
User-agent: *
Disallow: /

And I've also blocked some bad bot-crawlers on an .htaccess File level. For Apache::

Step 2

# Block specific bots by User-Agent
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DotBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SemrushBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule ^.* - [F,L]

Step 3

I've also enabled free version of Cloudfare.

Wellwisher

Solution above has fixed the problem for me.