How do I completely remove the horizontal scrollbars?

This is setting that is controlled by the OS. Please check http://osxdaily.com/2011/08/03/show-scroll-bars-mac-os-x-lion

In that case, they should be hidden while i'm not scrolling. Instead, they are always visible, including the vertical ones, because I've set them to  "When scrolling".

However, it might be worth mentioning that I'm using Material Theme Ui: 
https://plugins.jetbrains.com/plugin/8006-material-theme-ui

How scrollbars behave in other applications?

It also make sense to check with Material Theme UI plugin disabled.

There still doesn't seem to be an option to hide the horizontal scrollbars : /
I find no use for them and at the same time they are major visual distraction. Especially in Zen mode.

Daniel Popesky , what's this theme called?