I did this yesterday.
Tried running my own mail server first.
Couldn't get it to work with 587/STARTTLS even though the port it open.
Then I went for MailGun instead, still couldn't get it to work with 587/STARTTLS, however 465/SSL works just fine now. Not sure if there's something with my setup or Flarum that's causing 587 not to work.
Tag me and I'll gladly try to help you out, I'm quite active and fast at replying. ?
Tought, as 0E800 said, need to know which service you're trying to use.