All input becomes laggy when running in debug and open the console Follow

On Windows 10, when I am running a script using the debugger and I stop at a break point and open the console, for as long as that console is open all input to any open application on my system becomes incredibly laggy (to the point where it is totally unusable).

This is happening even when overall CPU / Memory usage remains low, e.g. 20% CPU, no more than 50% memory usage. And this happens regardless of if I disable all plugins.

Edit:- I have isolated the problem to if you ever import any iPython packages e.g. 

from IPython.display import SVG, display

Even if they aren't used in the script.