For anyone interested, I added the following CSS to custom styles.
a[href*="//"]:not([href*="mysite.com"]) {
position: relative;
}
a[href*="//"]:not([href*="mysite.com"]):after {
content: "\f35d";
font-family: "Font Awesome 5 Free";
position: relative;
top: -.4em;
padding: 0 .4em;
font-size: 70%;
}
Props to David Walsh for the selector.
Edited (15 Apr 2019) to call proper font-family and fix broken font awesome icon.