FoF Sitemap

[deleted]

clarkwinkelmann if you disable public user listing via Flarum permissions, they should also disapear from the sitemap.

It is the right fix, thank you.

[deleted]

clarkwinkelmann Hi, when if disabled user listing it caused permission error. This is a known error. https://discuss.flarum.org/d/18375-users-can-t-use-mention

When I set members to view user list, still users don't appear in sitemap although the SitemapGenerator.php file is unchanged. This is what I want, but I don't understand why users don't appear again on the site map.

clarkwinkelmann

[deleted] if you had a look at the source code (as it appears you have), you will notice a lot of "Guest" keywords in there.

It's because I'm using Flarum permission system to only return records (users and discussions) that would be visible to a guest. The fact you are logged in is ignored by the sitemap, as it's only designed to be returned to non-logged crawlers.

You will also notice that user profiles are not accessible either if you log out when setting the user list to members. So the sitemap reflects that.

[deleted]

clarkwinkelmann I set to permission to everybody, users appears in sitemap again, thanks for your detailed description.

iosmall

What if I have more than 100,000 URL links?

clarkwinkelmann

iosmall possibly some search engines will not accept the sitemap. I have never tested with so many links so I don't really know what might happen. The extension itself shouldn't break.

You may contract me to add the pagination feature if you need.

Ralkage

iosmall another thing I picked up in UI Design recently:

Sitemaps should be no larger than 50MB (52,428,800 bytes) and can contain a maximum of 50,000 URLs.

Please see: https://www.sitemaps.org/faq.html#faq_sitemap_size

Ralkage

Recently (yesterday) got certified in UI Design and sitemap came up A LOT.

Until recently I never understood the use of sitemaps because I rarely ever saw them on the variety of websites I generally visited and now I understand why they exist, to its entirety (not to be confused with a sitemap diagram lol).

There's a lot of things I learned but this extension really helps me understand sitemaps more, more so I wanted something Flarum related to being my first crack at one.

Thanks, @clarkwinkelmann ! 😃

matteocontrini

Ralkage but you can still break it in multiple parts 🙂

clarkwinkelmann

As I said before

clarkwinkelmann If there's interest I can implement the links limit (number and file size) according to the sitemap specs. The specs says there shouldn't be more than 50k links per sitemap file. For now it will keep growing forever. I'm not sure what is the actual search engine behavior if you exceed 50k links per file.

The only reason it isn't implemented is because the extension was contracted, and after consideration we didn't include this feature in the requirements. It shouldn't be too complicated to add, but can easily be one more hour of work, or more if we want to do proper testing. Feel free to reach out to me if you want this feature developed.

Leanna

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /www/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 158
I don't know how to solve this problem.

clarkwinkelmann

Leanna do you have a large number of posts/discussions/users ?

This extension fetches the content of the whole forum at once, so there's definitely a limit of the amount of data that can be queried that way. You might be the first person to encounter it 😅

We might be able to add a chunk query in there to improve the situation.

Leanna

clarkwinkelmann chunk

Leanna
I have limited ability and have not been able to solve this problem so far.

Leanna

clarkwinkelmann I have 100,000 forum content, 300,000 users.

tankerkiller125

clarkwinkelmann We might be able to add a chunk query in there to improve the situation.

@clarkwinkelmann This might actually not be a great idea, sitemaps do have a size limit that Google/Yahoo will support, personally I would highly recommend creating a sitemap index and then creating separate sitemaps for tags, and discussions, with extra discussion sitemaps anytime you reach the sitemap limit. The exact details on this is found at https://support.google.com/webmasters/answer/75712?hl=en

<?xml version="1.0" encoding="UTF-8"?>
   <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <sitemap>
      <loc>http://www.example.com/sitemap-tags.xml</loc>
      <lastmod>2004-10-01T18:23:17+00:00</lastmod>
   </sitemap>
   <sitemap>
      <loc>http://www.example.com/sitemap-disucssions.xml</loc>
      <lastmod>2005-01-01</lastmod>
   </sitemap>
   <sitemap>
      <loc>http://www.example.com/sitemap-disucssions-1.xml</loc>
      <lastmod>2005-01-01</lastmod>
   </sitemap>
</sitemapindex>

The exact sitemap limit is 50,000 entries or 50MB whichever one comes first. Further information on size can be located here

JacquesLim

when I set up the sitemap extension, I got the 500 http error. What's wrong with my flarum?

clarkwinkelmann

JacquesLim can you get the full error message ? You can follow https://flarum.org/docs/troubleshoot.html to enable debug mode or check the logs.

No one has yet confirmed whether or not the extension works with beta 9, so there's a possibility it's not compatible.

[deleted]

clarkwinkelmann seems to work for me on beta 9 unless I'm mistaken

matteocontrini

I have a discussion that is deleted but it's being included in the sitemap. Is this a known bug perhaps?

clarkwinkelmann

matteocontrini can you confirm if all your deleted discussions are visible, or just a specific one ?

The way the list of discussion is created is by listing all discussions that would be visible to a guest, so it is supposed to hide all deleted discussions that way. Is the discussion properly hidden from the index page when logged out ?

matteocontrini

clarkwinkelmann actually it's not deleted but locked, I confused the icons. I'm still sleeping. Sorry 🙂

« Previous Page Next Page »