There's no way to give granular access to the admin panel. Either you are an admin (Group ID 1) and you can access it, or you're not an admin (Not group ID 1) and you can't access it.
Extensions that want to provide moderator settings should add them to the forum frontend and not admin frontend. That's why the User directory and Mailing extensions are on the forum side for example, that way you can let non-admins access it as well.
EDIT: though if you dive deep into Flarum extension system, you could still apply some restrictions. Like for example we disabled the ability to view and change email settings on our demos forums. But this requires extra code for every feature because by default all permissions are checked with assertAdmin().