I was over thinking this. I think if a user has never seen markdown before (or BBcode), something like the below works in combination with the handy toolbar.
This only lists some of the extremely common options (bold, italics, etc.) and is very brief because I envision this being in a handy pop up.
The “more” at the end of each paragraph could link to the full documentation.
Use Markdown or BBcode to add formatting. The toolbar will help you add common Markdown which involves adding certain special characters in front of or around your text, such as:
![alt text](image url), more.
BBcode is a common bracket style code that wraps your text, such as:
It’s hard to imagine users not knowing what markdown or BBcode is, but I’m sure there’s a lot of folks who don’t. It depends on the forum’s target audience.