The extension manager is just a frontend for Composer, it can be removed without any impact to the installed extensions. But it shouldn't make any difference to have the extension manager and not use it or not have it at all, because it only does something when interacted with.
If you tried to install extensions with the extension manager and they don't work, it's also possible that the extension managed failed mid-way during installation and left you with a broken extension. In that case removing the extension manager won't help, but you could force a re-install of the broken extension with either the extension manager or Composer via the command line after identifying the cause of the failed installation.
