how to deindex users profile pages from google? robots.txt

Hari

Hi, i would like to deindex all users profile pages from google. in robots.txt i need to add these lines right ?

domain.com/flarum/u/admin

i need to add exclude lines in subfolder robots text file or root robots file ?
should i add disallow: /flarum/u/ or disallow: /flarum/u/* ?

or is there any setting to restrict other users to access the profile page? I want to set it to own user or admin to access a individual profile page

in beta 13 profile pages were not indexed.

Hari

Hari Got it we should always put this off so sitemap extension will take care of everything.

this should not be in robots.txt file

User-agent: *
Allow: /

robots txt should only have sitemap URL and nothing else

Sitemap: https://www.domain.com/if flarum in sub folder/sitemap.xml

Justoverclock

i think u can do something like:
Disallow: /u/

[deleted]

Justoverclock I also did the same
You can also validate it here
Make sure you enter slug only not domain like this { u/iamtest }

Hari

Justoverclock no star ? * I have read in Google docs .. confused . I just put it to off mode. I will wait for few days now see if user pages goes away from Google..

[deleted] if you suggesting to unlist using webmaster tools that's not a good idea Google says .

fakruzaruret

Justoverclock Google trying to index and give error on search console.

[deleted]

Hari no star ? * I have read in Google docs .. confused . I just put it to off mode. I will wait for few days now see if user pages goes away from Google..

You can play here by entering urls....

URL that starts with domain.yes/u/

URL that starts with domain.yes/discussionpage

Hari

[deleted] that's a testing page right .. I will try thanks

Hari

[deleted] that's a really cool tool and it helped, I have updated the robots txt. thanks 👍