Hari the reason why you need to hit that uninstall button is so that the migration down methods can run and drop the tables you no longer want or need that are associated with the extension you’re uninstalling 🙂
If you just do a straight composer remove, you risk having orphan data left over and risk running into issues reinstalling that same app extension later on.
