Debugging with pytest via tox using an interpreter from docker-compose does not respect breakpoints

Using the Python plugin in IntelliJ (Ultimate 2018.2 on Ubuntu, fully updated plugins etc), I have my python project setup to work within docker (docker-compose). 

Everything is fine there, I can setup a flask server and debug, I can setup other tasks/script with python and debug, all is fine using the docker interpreter.

We use pytest via tox for testing, and I setup a tox configuration using the default project interpreter (the one that lives in the docker container). It runs the tests just fine, everything looks great.

I have a tests that fails, so I add a breakpoint and click debug using the same tox setup, now either 1 of 2 things happen, either the tests all run but they run to the end ignoring my breakpoint OR the tests start to run, everything is setup, but then it just sits there frozen before actually running any tests. In the second case here, this sometimes happens regardless of if I've set a breakpoint or not.

How do I debug tests in my container if I cannot use the tox setup? Is there something I'm missing?

Thanks.