That's a nice project and I don't want to discourage you, but that sounds like a very complicated extension to start with!
If that's your first extension I would highly recommend creating a simpler extension first to get a hang of the Flarum extension system before going to something more complex. It might also make it easier for us to assist with new developer questions 🙂
The biggest challenge I see with tree-based comments is the layout. If you want to show the tree of comments reddit-style, you will need to rewrite the whole PostStream system used by Flarum to account for multiple streams loaded in each others, as well as rewrite all the infinite scrolling behavior that won't work with that new system. The stream system that ships with Flarum is only designed to show one continuous stream of chronological posts in a discussion.
If you just want to make a slightly different mention extension that still resembles the Flarum mention extension with additional color cues and/or shortcuts/filters, that's already much easier than something tree-based.
There are some existing discussions about a tree timeline here https://discuss.flarum.org/d/20766-alternate-tree-based-timeline