About don't write sitemap to the disk is a good idea, i love it. Thinking about big sites, add the file to the cache is the way to go... because query can be simple, but big sites can output easily more than 5 MB of data with a sitemap.

The feature that i miss a lot is have more control over parts of the board added to the sitemap (users, discussions and tags). Will be nice have a an option to include/exclude these three parts of the forum. Imagine that i decide not index tags, at this momet the plugin is limited because if i added a robots.txt to not allow indexing tags but i'm sending a sitemap that contain tag's urls, this is a confusing signal to the search engines.

Also be able to setup the "changefreq" and "priority" parameters according to the forum part will be great.

Also observed that:

  • On title changes the lastmod is not changed ❌
  • On reply added the lastmod is changed ✅
  • On edit contents of a post into discussion, the lastmod is not changed ❌

The Google's behavior with sitemaps bigger than 50k urls or 10MB is simply don't process them.

Hope that post this here also works instead go to GitHub ?

16 days later
a month later

tlalok is it Flarum's 404 page (with the error pages extension) or the webserver 404 page ?

If it's the webserver error page, make sure the /sitemap.xml url is handled by Flarum's index.php via htaccess or rewrite rules (should be with provided htaccess and example config). Maybe there are special rules in some hostings that prevent this file from being dynamically generated.

If it's Flarum 404 page, the extension is likely not enabled correctly.

You may open an issue at https://github.com/flagrow/sitemap/issues if you need further help ?

      tlalok Try disabling & re-enabling the extension.

        clarkwinkelmann Indeed, I just added this nginx directive and it works.

        location = /sitemap.xml { try_files $uri $uri/ /index.php?$query_string; }

        clarkwinkelmann If it's the webserver error page, make sure the /sitemap.xml url is handled by Flarum's index.php via htaccess or rewrite rules (should be with provided htaccess and example config). Maybe there are special rules in some hostings that prevent this file from being dynamically generated.

        I use this docker image, so I am going to fix this issue.

        @clarkwinkelmann @arda thanks for your help ?

              2 months later
              8 days later

              Updated for beta 8.

              I've kept compatibility with the Pages extension for now, even though it hasn't been updated for beta 8 yet. Be warned that an additional update of this extension might be needed to support the future beta8-compatible version of Pages.

                clarkwinkelmann Thanks for the update, unfortunately it doesn't seem to work at my end. When accessing url/sitemap.xml I'm only getting a 404, both with the extension enabled and disabled. I'm using nginx as my webserver.

                    Kakifrucht Could this be a similar case to tlalok where the webserver needs an additional rewrite rule to handle .xml files via the PHP application ? Maybe try the solution above to see if it solves your case.

                    Also is the 404 from Flarum (there should be a back to homepage link) or Nginx (just a 404 message in english) ? If nginx is responding, then it has something to do with server configuration (and it seems the fix above solve it).

                    If it's a Flarum 404 error page then we'll have to investigate further. Could you open an issue at GitHub if that's the case so we don't create too much noise here ?

                          clarkwinkelmann That's what I was looking for, sorry for being too lazy to scroll a couple of posts up.

                          You might want to add it to the installation section of this discussion for nginx users.

                              Kakifrucht You might want to add it to the installation section of this discussion for nginx users.

                              Yes indeed, I'll do that. Though it doesn't seem to apply to everybody. I didn't get any feedback of anybody getting it working on nginx, but I'm running the extension on my own nginx server and didn't need the additional config.

                              I should try to narrow it down to identify what's the cause for this, but I don't have the time 💩

                                  clarkwinkelmann but I'm running the extension on my own nginx server and didn't need the additional config.

                                  I really only followed the install instructions for nginx from here.

                                  This is what my server configuration block looks like:

                                  server {
    listen 80;
    server_name myurl.com;
    root path/to/flarum/public;
    include path/to/flarum/.nginx.conf;
    
    location = /sitemap.xml {
        try_files $uri $uri/ /index.php?$query_string;
    }
    
    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/run/php/php7.2-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
    }
}

                                  I then used certbot to apply a SSL certificate and set it to redirect everything to https.

                                    3 months later

                                    • [deleted]

                                    • Post #24
                                    • Edited

                                    FriendsOfFlarum How can i remove users in sitemap.

                                    <loc>https://mysite.com/u/user1</loc>
<lastmod>2019-03-25T18:04:08+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>0.5</priority>
<url>
<loc>https://mysite.com/u/user2</loc>
<lastmod>2019-03-25T18:04:08+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>0.5</priority>

                                    Solved: Open SitemapGenerator.php in vendor/flagrow/sitemap/src

                                    Remove lines:

                                    $users = User::whereVisibleTo(new Guest())->get();

    foreach ($users as $user) {
        $urlSet->addUrl($url . '/u/' . $user->username, Carbon::now(), Frequency::DAILY, 0.5);
    }

                                        [deleted] this should only be considered a temporary solution because next time you run composer those files might be reverted to their original state.

                                        The sitemap extension is permission based, so if you disable public user listing via Flarum permissions, they should also disapear from the sitemap.

                                        There is no manual options at this time. The idea is that any publicly visible page should be in the sitemap.

                                            I've just identified an issue with SEO friendly slugs that can cause Sitemap to break if you're using both extension and that somebody inputs % in a discussion title.

                                            Because you can't prevent users from doing that and because it might prevent search engines from reading the XML file, I strongly recommend not installing/enabling SEO Friendly slugs with the sitemap.

                                            FoF Transliterator appears to cover those edge cases, so I'd recommend using that extension instead for friendly slugs in non-english.

                                              • [deleted]

                                              • Post #27
                                              • Edited

                                              clarkwinkelmann if you disable public user listing via Flarum permissions, they should also disapear from the sitemap.

                                              It is the right fix, thank you.