Pycharm debugger takes a long time to stop when using SSH deployment

It takes about half to one minute each time to stop the debugger or rerun it.

I guess the problem is not with the SSH connection as the debugger starts immediately - the delay only happens at stopping the debugger.

Which is even more interesting is that if the debugger stops due to an exception that will lead to an error, continuing the debugger with F9 also happen to finish the process immediately. However, if I try to Stop debugging with Ctrl+F2, it takes a long time again.

It's very annoying, is there any idea why this happens, or some workaround to get rid of this?

Thank you!