crazysvn it sounds like this extension is exactly what you need. Is it a problem to have more information? That's the point of an audit log, better have too much logged than not enough.
The free version only shows all logs in a single list, which you can filter by s specific user with a gambit. The pro version offers a convenient access to a users actions from their profile page.
If that's useful to you, I could add a feature in the pro version to show the actions of all users belonging to a specific group in the global list.
You could also use the existing action gambit to only show actions that are restricted to moderators, then you wouldn't even really need the group filter unless you have a lot of actions by admins and only want to see mods.
I believe there is another audit log extension by another developer as well, but I don't have the link right now.