The code we use resizes the image to be maximum 60 pixels tall, and saves it as PNG. It should preserve transparency https://github.com/flarum/framework/blob/main/framework/core/src/Api/Controller/UploadLogoController.php
Depending on what is available on your system, it might use either Imagick or GD, so transparency support in various file type could be different.
If nothing works, you can manually resize your logo to be 60 pixels tall and replace the broken logo file which you will find in <flarum>/public/assets/logo-<random number>.png
If you want to use a different file format (like gif, jpg or svg) you can edit the filename which is saved in the settings table of the database. Using SVG might require adding some custom CSS as well to ensure correct size.
EDIT: I realize now you were probably referring to the favicon. In this case it's still a very similar logic. Flarum resizes the image to 64x64 and encodes to PNG in https://github.com/flarum/framework/blob/main/framework/core/src/Api/Controller/UploadFaviconController.php . The filename in assets starts with favicon-.
