Paging link & Crawlers

martijn-benjamin

Hi guys,

Congrats on the work so far! Looks promising.

Something i bumped into: I was creating a sitemap and found that the paging link will keep crawlers very busy.

Regardless of actual pages the paging link will be in the page source:

In the standard install the paging link is hidden when not applicable but a crawler will follow it and end up going to page 2 where it finds the paging link to page 3 and so on infinitely, which is a long time ;0)

My solution for now is to exclude the url in robots.txt but that would for now mean posts will not be indexed which is a sad thing.

Or am i missing something here?

Cheers!
Martijn

martijn-benjamin

I'm guessing this is no so much of an issue?

Digital

Apologies for such delay. I will get in touch with our experts to help you with this issue.

martijn-benjamin

Cheers mate!

Kulga

yep, happened to me as well several times. For some reason though, can't find the logs - but I believe it's gone as high as page 8000
There's nothing you can really do right now as there is no difference (that I can tell) between a successful result and a empty one. So best things to do are

Choose whether to block it completely (as OP mentioned) or not. This probably isn't needed as the cpu usage isn't that high (very annoying for caching though!)
Submit a bug report - https://github.com/flarum/core/issues

I suppose to potentially resolve this, not having a "next" paging link and/or having a 404 response when there is no result.

martijn-benjamin

Thnx Kulga.

I'm wondering though... how is the paging used actually? There is an infinite scroll into the depth of the page, so when would i need the paging link? Or are these links used for loading the messages on scrolling?

Kulga

martijn-benjamin There is an infinite scroll into the depth of the page, so when would i need the paging link?

For you, probably not for non-javascript users and crawlers.

martijn-benjamin

Ah yes, i see.

And it's the crawlers that will have trouble with this thing...

I will submit the bug report ;0)