0E800 It should, though?
Clicking down vote after clicking up vote is like changing your vote, so it should take away the +1 you added (assuming there are no other votes on the post, it would now be 0) and add -1 (which makes the total now -1).
If you unclick the up vote, it goes back to 0, same for the down vote. If you click the opposite vote, it behaves as above.
ETA:
Kyrne Added the ability for people to see who upvoted/downvoted a post configurable via permission
Can you add some kind of change on hover to the vote count number when someone has this permission? It has pretty low discoverability. Maybe some descriptive text on hover (click to view votes?) or a CSS style change?