si_edgey as far as I know, two options exist: customizing the page not found error message with Linguist extension to say something different, or use the custom HTML error pages to completely change the template of the 404 page.
Due to how Flarum is implemented, there's no way to open the login modal directly on error page. With the Direct Links extension you could have a link that can be pasted in the custom HTML error page.
More advanced solutions (like intentionally leaking the content type or existence status of the discussion to have a custom error message only for private discussions) would require developing a custom extension.