SEO and robot.txt optimization for Flarums

Zeokat

The next code should be enougth to not index user profiles from my experience.

User-agent: *
Disallow: /u/

Sitemap: https://www.seekadventure.net/sitemap-post.xml

At this stage i don't add more rules to robots because i hope that some issues will be fixed on future.

MikeJones

Zeokat So I don't need a the /u/* for everything after the u? and you do not think I need to not index the /d/d?

Zeokat

With Disallow: /u/, robots can't access to the /u/ folder, so... who cares what goes after the /u/.

About your second question, i don't know what kind of pages are those that you name as /d/d. The location /d/ is reserved to discussions and i can't see a proper reason to not index them...

MikeJones

Zeokat you are correct about the /d/ I just double checked.

What I am talking about trying to disallow is each comment has its own link and is thus getting indexed twice. Once at the discussion level and once for its individual comment.

If you click the time next to you name it provides an individual link directly to your comment like this:

https://discuss.flarum.org/d/13383-seo-and-robot-txt-optimization-for-flarums/4

Zeokat

MikeJones That is something that should be fixed at core level (adding proper canonical URLs), not with a robots.txt file.

Pollux

MikeJones

Two things to mention:

The name of the file is robots.txt, with an "s" in the middle.
The link to your sitemap is missing two slashes after https:

MikeJones

Pollux Thanks, I noticed the robot"s" issue when I tried to have google find it lol, they kept telling me they couldn't find it.... so that's fixed... Going to fix the sitemap link now. Thanks!

Edit: All should be good now: https://www.seekadventure.net/robots.txt

Aleram

MikeJones So why did you finally give up the idea to prohibit indexing /u/ ?

nitaaikumar

Aleram Disallow: /u/ I have added this. Google search console is sending me these Coverage errors because I guess the user profiles are submitted by the sitemap generated via the Flarum SEO extension.

Search Console has identified that your site is affected by 1 Coverage issues:

Top Errors

Errors can prevent your page or feature from appearing in Search results. The following errors were found on your site:

Submitted URL blocked by robots.txt

We recommend that you fix these issues when possible to enable the best experience and coverage in Google Search.

I still want to keep disallow /u/. So a bit of dilemma.

bluisier

@MikeJones I'm setting up a Flarum forum ended here when trying to remove the users pages with /u/ from the sitemap. I couldn't resist checking your Sitemap out and I saw there are no users urls in it. How did you manage to do it?

As a side note, from what I understand of Sitemaps and robots.txt, they do not prevent your pages to be indexed (more to be crawled). The best solution I guess would be to add a <meta name="robots" content="noindex"> on all user pages. I imagine that could be a nice add to the SEO extension

MikeJones

bluisier I don't think I did anything specific. Maybe the sitemap extension doesn't add user urls...not sure though

bluisier

MikeJones if you want to check, I came up with another solution to avoid indexing some pages in google (like users or tag pages)... Here it is.

cmwetherell

@MikeJones it looks like your current sitemap does not have the disallow /u/:

User-agent: *
Allow: /

Sitemap: https://www.seekadventure.net/sitemap.xml

My sitemap.xml generated by the SEO extension includes all of the /u/ URLs, but yours does not. I am also using the sitemap extension. If you figure out why that is the case, please let me know!