The FOF/Sitemap extension's readme suggests you are able to remove specifics from being pushed into the sitemap
Remove a Resource
In a very similar way, you can also remove resources from the sitemap:
return [
(new \FoF\Sitemap\Extend\RemoveResource(\FoF\Sitemap\Resources\User::class)),
];
I'm not a developer but could be the right avenue for you to wander down.