Math expressions need to be escaped because they are parsed by the Less compiler by default, even if your source file extension is .css
or using the admin custom styles.
See https://stackoverflow.com/questions/17904088/disable-less-css-overwriting-calc for the syntax.
For example width: ~"calc(100% - 200px)";
I thought I had written a discussion about this in the past but it doesn't seem like it. At some point I had suggested Flarum uses the StrictMath option by default but this was never changed https://discuss.flarum.org/d/4882-enable-strictmath-for-less-compilation.
Something to keep in mind when searching help for the Less syntax/compiler is that Flarum uses less.php
which doesn't support all options of native Less https://github.com/wikimedia/less.php
I hope that helps!