On small forums, it won't really matter. Any performance impact will be unnoticeable.
If you remove an extension and know you don't plan to re-enable it in the future, it makes sense to remove the extension's database modifications and data to improve performance and reduce risk of a conflict with a different extension in the future if some names are re-used between extensions.
You should avoid modifying the database manually though. Flarum has a built-in feature called "Purge" that you can access on a disabled extension's page in the admin panel. This button will do this task for you, and delete any database table, column and data belonging to the extension in a clean manner. If you already uninstalled an extension, you can re-install it, then visit its disabled page in the admin panel to perform the Purge action. You don't need to re-enable the extension.
If you encounter an error while purging an extension, look for the error message in Flarum log file, and share it on the extension's discussion on the forum. If an error happens, a manual cleanup might be necessary. The community can provide guidance on a case-by-case basis if this happens. While it's rare, there's always a possibility that there is a bug in the purge code of an extension, and because very few people use the Purge feature, a bug can sometimes go unnoticed for a long time. I mention this because it has happened a few times before.
It's always a good idea to back up the forum before performing the purge.
