FoF Sitemap

Zeokat

About don't write sitemap to the disk is a good idea, i love it. Thinking about big sites, add the file to the cache is the way to go... because query can be simple, but big sites can output easily more than 5 MB of data with a sitemap.

The feature that i miss a lot is have more control over parts of the board added to the sitemap (users, discussions and tags). Will be nice have a an option to include/exclude these three parts of the forum. Imagine that i decide not index tags, at this momet the plugin is limited because if i added a robots.txt to not allow indexing tags but i'm sending a sitemap that contain tag's urls, this is a confusing signal to the search engines.

Also be able to setup the "changefreq" and "priority" parameters according to the forum part will be great.

Also observed that:

On title changes the lastmod is not changed ❌
On reply added the lastmod is changed ✅
On edit contents of a post into discussion, the lastmod is not changed ❌

The Google's behavior with sitemaps bigger than 50k urls or 10MB is simply don't process them.

Hope that post this here also works instead go to GitHub ?

MikeJones

Just installed, excited to have it up and running! Thanks again!

tlalok

Hi,

I just installed it but url https://mondedie.fr/sitemap.xml does not work. I receive a 404 error.
Any idea?

clarkwinkelmann

tlalok is it Flarum's 404 page (with the error pages extension) or the webserver 404 page ?

If it's the webserver error page, make sure the /sitemap.xml url is handled by Flarum's index.php via htaccess or rewrite rules (should be with provided htaccess and example config). Maybe there are special rules in some hostings that prevent this file from being dynamically generated.

If it's Flarum 404 page, the extension is likely not enabled correctly.

You may open an issue at https://github.com/flagrow/sitemap/issues if you need further help ?

arda

tlalok Try disabling & re-enabling the extension.

tlalok

clarkwinkelmann Indeed, I just added this nginx directive and it works.

location = /sitemap.xml { try_files $uri $uri/ /index.php?$query_string; }

clarkwinkelmann If it's the webserver error page, make sure the /sitemap.xml url is handled by Flarum's index.php via htaccess or rewrite rules (should be with provided htaccess and example config). Maybe there are special rules in some hostings that prevent this file from being dynamically generated.

I use this docker image, so I am going to fix this issue.

@clarkwinkelmann @arda thanks for your help ?

clarkwinkelmann

Kakifrucht Could this be a similar case to tlalok where the webserver needs an additional rewrite rule to handle .xml files via the PHP application ? Maybe try the solution above to see if it solves your case.

Also is the 404 from Flarum (there should be a back to homepage link) or Nginx (just a 404 message in english) ? If nginx is responding, then it has something to do with server configuration (and it seems the fix above solve it).

If it's a Flarum 404 error page then we'll have to investigate further. Could you open an issue at GitHub if that's the case so we don't create too much noise here ?

niions

Does it work with beta 8? I get an empty sitemap response.

clarkwinkelmann

niions not compatible yet. Will be updated soon ™️

niions

clarkwinkelmann thankyou!

clarkwinkelmann

Updated for beta 8.

I've kept compatibility with the Pages extension for now, even though it hasn't been updated for beta 8 yet. Be warned that an additional update of this extension might be needed to support the future beta8-compatible version of Pages.

Kakifrucht

clarkwinkelmann Thanks for the update, unfortunately it doesn't seem to work at my end. When accessing url/sitemap.xml I'm only getting a 404, both with the extension enabled and disabled. I'm using nginx as my webserver.

Kakifrucht

clarkwinkelmann That's what I was looking for, sorry for being too lazy to scroll a couple of posts up.

You might want to add it to the installation section of this discussion for nginx users.

clarkwinkelmann

Kakifrucht You might want to add it to the installation section of this discussion for nginx users.

Yes indeed, I'll do that. Though it doesn't seem to apply to everybody. I didn't get any feedback of anybody getting it working on nginx, but I'm running the extension on my own nginx server and didn't need the additional config.

I should try to narrow it down to identify what's the cause for this, but I don't have the time 💩

Kakifrucht

clarkwinkelmann but I'm running the extension on my own nginx server and didn't need the additional config.

I really only followed the install instructions for nginx from here.

This is what my server configuration block looks like:

server {
    listen 80;
    server_name myurl.com;
    root path/to/flarum/public;
    include path/to/flarum/.nginx.conf;
    
    location = /sitemap.xml {
        try_files $uri $uri/ /index.php?$query_string;
    }
    
    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/run/php/php7.2-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
    }
}

I then used certbot to apply a SSL certificate and set it to redirect everything to https.

[deleted]

FriendsOfFlarum How can i remove users in sitemap.

<loc>https://mysite.com/u/user1</loc>
<lastmod>2019-03-25T18:04:08+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>0.5</priority>
<url>
<loc>https://mysite.com/u/user2</loc>
<lastmod>2019-03-25T18:04:08+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>0.5</priority>

Solved: Open SitemapGenerator.php in vendor/flagrow/sitemap/src

Remove lines:

$users = User::whereVisibleTo(new Guest())->get();

    foreach ($users as $user) {
        $urlSet->addUrl($url . '/u/' . $user->username, Carbon::now(), Frequency::DAILY, 0.5);
    }

clarkwinkelmann

[deleted] this should only be considered a temporary solution because next time you run composer those files might be reverted to their original state.

The sitemap extension is permission based, so if you disable public user listing via Flarum permissions, they should also disapear from the sitemap.

There is no manual options at this time. The idea is that any publicly visible page should be in the sitemap.

[deleted]

clarkwinkelmann if you disable public user listing via Flarum permissions, they should also disapear from the sitemap.

It is the right fix, thank you.

[deleted]

clarkwinkelmann Hi, when if disabled user listing it caused permission error. This is a known error. https://discuss.flarum.org/d/18375-users-can-t-use-mention

When I set members to view user list, still users don't appear in sitemap although the SitemapGenerator.php file is unchanged. This is what I want, but I don't understand why users don't appear again on the site map.

clarkwinkelmann

I've just identified an issue with SEO friendly slugs that can cause Sitemap to break if you're using both extension and that somebody inputs % in a discussion title.

Because you can't prevent users from doing that and because it might prevent search engines from reading the XML file, I strongly recommend not installing/enabling SEO Friendly slugs with the sitemap.

FoF Transliterator appears to cover those edge cases, so I'd recommend using that extension instead for friendly slugs in non-english.

FBI

clarkwinkelmann Thanks for info.
I've found thousands from my forum 😃

clarkwinkelmann

[deleted] if you had a look at the source code (as it appears you have), you will notice a lot of "Guest" keywords in there.

It's because I'm using Flarum permission system to only return records (users and discussions) that would be visible to a guest. The fact you are logged in is ignored by the sitemap, as it's only designed to be returned to non-logged crawlers.

You will also notice that user profiles are not accessible either if you log out when setting the user list to members. So the sitemap reflects that.