I don't think we have any example of an extension that opens a modal that can insert content back into the post in return.
The closest examples could be the Upload extension and Emoji picker probably.
Any extension that adds a button to the toolbar will be a good example for adding the button. Then that button could open a modal and pass the editor state as an attr. Then the modal can be based on any other modal found in any extension, with its submit button calling the insert text method on the editor state before closing.