UaMV the problem is that I forced the actor (logged in user) as a recipient if there are no recipient groups and the actor was not an recipient. This was done when saving the recipients. It makes more sense to add the current user to the recipients popup box, allowing the user to remove himself if he deems necessary. That way I won't need to force the actor as recipient anymore and it would allow removing yourself too.
This was fixed in development.