meta "noindex" for user or tag pages

bluisier

Hi everybody,

I'm new to Flarum and first I would like to thank all the people that contribute to Flarum in some ways... Flarum is already super nice and I can't wait to see how it's going to grow in the future.

I was reading a few discussion here and here about sitemap or robot.txt to avoid indexing user pages (or other pages that have not a lot of content).

From what I understand (from google doc), the proper way of not to index some pages is to add a tag "noindex" in the head of pages that should not be indexed. So I have 2 questions about that:

Is it possible to do that in Flarum already? (I haven't found any solution but maybe I just missed it because I'm new to Flarum)
If not, is it smth that should be implemented in an existing extension (like the SEO one) or does it deserve its own extension? (I'm not afraid of coding smth if necessary)

Thanks a lot for your help

clarkwinkelmann

bluisier welcome to the project!

Just like the SEO extension adds meta head tags to the page, you could very well do the same for nofollow head tag from a separate extension.

One limitation in the current Flarum version this requires inspecting the page URL to "guess" which page you're on (that's what the SEO extension does). In the future release extensions will be able to do the same based on route name, which will be even more reliable since it will continue to work even if the URL are customized.

As for including in an extension or creating a new one, both are possible. It's usually a matter of whether the developers prefer to pack many features together, or if they prefer to separate them. Personally, I'm a fan of making as many small extensions as possible, because this allows people to only install what they really need, and forces developers to decouple a feature from other unrelated features.

bluisier

clarkwinkelmann thanks for your answer!

Before I start digging into flarum extensions, from what I could read here, it should be possible to avoid indexing pages by adding smthing like this in my nginx config file (in a specific location block like this for exemple):

location /forum/u {
    add_header  X-Robots-Tag "noindex";
    try_files $uri $uri/ /forum/index.php?$query_string;
}

I tried it but somehow I couldn't see the X-Robots-Tag in my network tab... I'm missing smth?

clarkwinkelmann

bluisier are you sure the location blocks are in an order where this one will be picked up in priority over the root location? Have you restarted nginx?

I've never tried adding headers to specific routes with nginx. Maybe there's another better syntax as to not duplicate the try_files statement.

bluisier

clarkwinkelmann so after playing around for a bit, I have a working solution that allows me to control which pages should be indexed in google and which one should not (without building an extension). So as the Google documentation specify, the idea is to add the header X-Robots-Tag: noindex to the pages that should not be indexed.

With nginx, I did it as follow. In my http block, I added:

map $request_uri $robots_header {
        default "";
        ~^/forum/u(.*) "noindex";
        ~^/forum/t(.*) "noindex";
    }

In my server block, I added then the following line:

more_set_headers "X-Robots-Tag: $robots_header";

Note that because I have a setup with an nginx proxy in front of nginx, I had to use the headers more module. If you don't have an nginx proxy, you can probably just add the following line in your server block:

add_header X-Robots-Tag $robots_header;

That's the cleanest solution I could come up with 🙂 . Let's see if google take the header into account and exclude theses urls in my google search console