[PyCharm Community 2019.1.3] Debug view for pytest freeze and can't access variables

When I wants to debug some pytest, the view appear, the app is working, I can set breakpoint and step into it.

Except I can't see variables values, I cant' access a debug console with the proper context.

There is something looping forever somewhere, see attached file.

Any idea?

12 comments
Comment actions Permalink

Ok, glad to hear it's working. If the issue reproduces again, you can try disabling cython speedups, which can cause peformance issues in rare cases. To do that, set the following env variables to your test configuration:

PYDEVD_USE_CYTHON=NO
PYDEVD_USE_FRAME_EVAL=NO

1
Comment actions Permalink

Does the issue reproduce with any test script? Please try some simple example, to determine if it's related to your scripts.

Then, please set `PYCHARM_DEBUG=True` env variable in your pytest configuration, run the debugger - it should show the extended output in the console, please paste here.

0
Comment actions Permalink

When trying to reproduce, I created a new project dedicated to this issue.

And I remember that my original source was on a project that is mainly a Java app, with this folder containing some python helpers.

So I modified the original project to have a separate Python env and just imported pytest (and boto3 because it is a AWS helper script)

And now everything works.

I'll try later to reproduce in another project, using my system python env. Maybe there is something wrong in it.

But I have a solution to my initial issue.

0
Comment actions Permalink

I have the same issue on PyCharn pro 2020.2. Starting with a fresh project and a very simple dummy test:

def test_var():

t = 1
assert t == 1

Setting a breakpoint works, the program hold, but PyCharm can't resolve variables (hangs on "Collecting data...") and the console won't print. Here is the console output when trying to do things:

 

/home/mowglie/.pyenv/versions/py3/bin/python /home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 41727 --file /home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/pycharm/_jb_pytest_runner.py --target test_dummy.py::test_var -- -s
Executing PyCharm's `sitecustomize`
User doesn't have a custom `sitecustomize`
Custom matplotlib backend was set for SciView
Using Cython speedups
Unable to load jupyter_debug plugin
Executing file /home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/pycharm/_jb_pytest_runner.py
arguments: ['/home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/pycharm/_jb_pytest_runner.py', '--target', 'test_dummy.py::test_var', '--', '-s']
PYDEVD_FILTER_LIBRARIES False
Started in multiproc mode
('Connecting to ', '127.0.0.1', ':', '41727')
('Connected.',)
debugger: received >>99 -1 35547
<<
Received command: ??? 99 -1 35547
Received port 35547
pydev debugger: process 12882 is connecting
('Connecting to ', '127.0.0.1', ':', '35547')
('Connected.',)
debugger: received >>501 1 0.1 UNIX
<<
Received command: CMD_VERSION 501 1 0.1 UNIX
sending cmd --> CMD_VERSION @@BUILD_NUMBER@@
Connected to pydev debugger (build 202.7319.64)
debugger: received >>111 3 python-line /home/mowglie/PycharmProjects/pythonProject/test_dummy.py 5 test_var THREAD None None
<<
Received command: CMD_SET_BREAK 111 3 python-line /home/mowglie/PycharmProjects/pythonProject/test_dummy.py 5 test_var THREAD None None
Added breakpoint:/home/mowglie/PycharmProjects/pythonProject/test_dummy.py - line:5 - func_name:b'test_var'
debugger: received >>170 11 SET_UNIT_TESTS_DEBUGGING_MODE
146 7 SHOW_RETURN_VALUES 0
101 9
122 5 python-BaseException None None 0 1 0
<<
Received command: ??? 170 11 SET_UNIT_TESTS_DEBUGGING_MODE
Received command: CMD_SHOW_RETURN_VALUES 146 7 SHOW_RETURN_VALUES 0
Show return values: False
Received command: CMD_RUN 101 9
Received command: CMD_ADD_EXCEPTION_BREAK 122 5 python-BaseException None None 0 1 0
Exceptions to hook on terminate: {'BaseException': <_pydevd_bundle.pydevd_breakpoints.ExceptionBreakpoint object at 0x7fdc06f38e80>}
sending cmd --> CMD_THREAD_CREATE <xml><thread name="MainThread" id="pid_12882_id_140583053184448" /></xml>
IDE_PROJECT_ROOTS ['/home/mowglie/PycharmProjects/pythonProject']
LIBRARY_ROOTS ['/home/mowglie/.pyenv/versions/3.8.5/lib/python3.8', '/home/mowglie/.pyenv/versions/3.8.5/lib/python3.8/lib-dynload', '/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages', '/home/mowglie/disk/Dropbox/HomeDocs/git/salem', '/home/mowglie/disk/Dropbox/HomeDocs/git/oggm-fork', '/home/mowglie/.cache/JetBrains/PyCharm2020.2/python_stubs/551278590', '/home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/python-skeletons', '/home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/typeshed/stdlib/3.7', '/home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/typeshed/stdlib/3', '/home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/typeshed/stdlib/2and3', '/home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/typeshed/third_party/3', '/home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/typeshed/third_party/2and3']
Launching pytest with arguments -s test_dummy.py::test_var in /home/mowglie/PycharmProjects/pythonProject
Patching args: ['/sbin/ldconfig', '-p']
Process is not python, returning.
sending cmd --> CMD_PROCESS_CREATED <process/>
=== Scientific View debug enabled ===
sending cmd --> CMD_PROCESS_CREATED <process/>
('Connecting to ', '127.0.0.1', ':', '41727')
('Connected.',)
("The debugger thread 'DispatchReader' is traced which may lead to debugging performance issues.",)
debugger: received >>99 -1 43425
<<
Received command: ??? 99 -1 43425
PYDEVD_FILTER_LIBRARIES False
('Connecting to ', '127.0.0.1', ':', '43425')
('Connected.',)
debugger: received >>501 1 0.1 UNIX
<<
Received command: CMD_VERSION 501 1 0.1 UNIX
sending cmd --> CMD_VERSION @@BUILD_NUMBER@@
debugger: received >>111 3 python-line /home/mowglie/PycharmProjects/pythonProject/test_dummy.py 5 test_var THREAD None None
<<
Received command: CMD_SET_BREAK 111 3 python-line /home/mowglie/PycharmProjects/pythonProject/test_dummy.py 5 test_var THREAD None None
debugger: received >>111 13 python-line /home/mowglie/PycharmProjects/pythonProject/test_dummy.py 5 test_var THREAD None None
<<
Received command: CMD_SET_BREAK 111 13 python-line /home/mowglie/PycharmProjects/pythonProject/test_dummy.py 5 test_var THREAD None None
Added breakpoint:/home/mowglie/PycharmProjects/pythonProject/test_dummy.py - line:5 - func_name:b'test_var'
debugger: received >>122 5 python-BaseException None None 0 1 0
<<
Received command: CMD_ADD_EXCEPTION_BREAK 122 5 python-BaseException None None 0 1 0
Added breakpoint:/home/mowglie/PycharmProjects/pythonProject/test_dummy.py - line:5 - func_name:b'test_var'
Exceptions to hook on terminate: {'BaseException': <_pydevd_bundle.pydevd_breakpoints.ExceptionBreakpoint object at 0x7fdbada823a0>}
debugger: received >>146 7 SHOW_RETURN_VALUES 0
170 11 SET_UNIT_TESTS_DEBUGGING_MODE
101 9
<<
Received command: CMD_SHOW_RETURN_VALUES 146 7 SHOW_RETURN_VALUES 0
Show return values: False
Received command: ??? 170 11 SET_UNIT_TESTS_DEBUGGING_MODE
Received command: CMD_RUN 101 9
debugger: received >>122 15 python-BaseException None None 0 1 0
170 17 SET_UNIT_TESTS_DEBUGGING_MODE
146 19 SHOW_RETURN_VALUES 0
<<
Received command: CMD_ADD_EXCEPTION_BREAK 122 15 python-BaseException None None 0 1 0
Exceptions to hook on terminate: {'BaseException': <_pydevd_bundle.pydevd_breakpoints.ExceptionBreakpoint object at 0x7fdbf64072e0>}
Received command: ??? 170 17 SET_UNIT_TESTS_DEBUGGING_MODE
Received command: CMD_SHOW_RETURN_VALUES 146 19 SHOW_RETURN_VALUES 0
Show return values: False
sending cmd --> CMD_THREAD_CREATE <xml><thread name="Thread-11" id="pid_12912_id_140581488043632" /></xml>
sending cmd --> CMD_THREAD_CREATE <xml><thread name="Thread-12" id="pid_12912_id_140581488045600" /></xml>
sending cmd --> CMD_THREAD_KILL pid_12912_id_140581488045600
sending cmd --> CMD_THREAD_CREATE <xml><thread name="Thread-13" id="pid_12912_id_140581488045696" /></xml>
sending cmd --> CMD_THREAD_KILL pid_12912_id_140581488045696
sending cmd --> CMD_THREAD_CREATE <xml><thread name="Thread-14" id="pid_12912_id_140581633657008" /></xml>
sending cmd --> CMD_THREAD_KILL pid_12912_id_140581633657008
sending cmd --> CMD_THREAD_CREATE <xml><thread name="MainThread" id="pid_12882_id_140583053184448" /></xml>
============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-6.0.2, py-1.9.0, pluggy-0.13.1 -- /home/mowglie/.pyenv/versions/py3/bin/python
cachedir: .pytest_cache
Matplotlib: 3.3.2
Freetype: 2.6.1
rootdir: /home/mowglie/PycharmProjects/pythonProject
plugins: mpl-0.122, oggm-1.3.2.dev95+gc05d330
collecting ... collected 1 item
test_dummy.py::test_var Suspending at breakpoint in file: /home/mowglie/PycharmProjects/pythonProject/test_dummy.py on line 5
sending cmd --> CMD_THREAD_SUSPEND <xml><thread id="pid_12882_id_140583053184448" stop_reason="111" message="" suspend_type="frame_eval"><frame id="140582185093536" name="test_var" file="/home/mowglie/PycharmProjects/pythonProject/test_dummy.py" line="5"></frame><frame id="140582185125248" name="pytest_pyfunc_call" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/python.py" line="180"></frame><frame id="94234090882784" name="_multicall" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/callers.py" line="187"></frame><frame id="140582185028848" name="&lt;lambda&gt;" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/manager.py" line="84"></frame><frame id="140582185028384" name="_hookexec" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/manager.py" line="93"></frame><frame id="140582185054272" name="__call__" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/hooks.py" line="286"></frame><frame id="140582185120640" name="runtest" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/python.py" line="1570"></frame><frame id="94234090877504" name="pytest_runtest_call" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/runner.py" line="153"></frame><frame id="94234090882144" name="_multicall" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/callers.py" line="187"></frame><frame id="140582185027920" name="&lt;lambda&gt;" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/manager.py" line="84"></frame><frame id="140582185027456" name="_hookexec" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/manager.py" line="93"></frame><frame id="140582185026992" name="__call__" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/hooks.py" line="286"></frame><frame id="140582185070144" name="&lt;lambda&gt;" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/runner.py" line="247"></frame><frame id="94234090804336" name="from_call" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/runner.py" line="294"></frame><frame id="140582185091136" name="call_runtest_hook" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/runner.py" line="246"></frame><frame id="140582185041248" name="call_and_report" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/runner.py" line="207"></frame><frame id="94234090878144" name="runtestprotocol" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/runner.py" line="117"></frame><frame id="140582185066560" name="pytest_runtest_protocol" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/runner.py" line="100"></frame><frame id="94234090876864" name="_multicall" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/callers.py" line="187"></frame><frame id="140582184987888" name="&lt;lambda&gt;" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/manager.py" line="84"></frame><frame id="94234090777440" name="_hookexec" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/manager.py" line="93"></frame><frame id="140582185300960" name="__call__" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/hooks.py" line="286"></frame><frame id="94234090730336" name="pytest_runtestloop" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/main.py" line="321"></frame><frame id="94234090876224" name="_multicall" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/callers.py" line="187"></frame><frame id="140582185078000" name="&lt;lambda&gt;" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/manager.py" line="84"></frame><frame id="140582185077536" name="_hookexec" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/manager.py" line="93"></frame><frame id="140582185077072" name="__call__" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/hooks.py" line="286"></frame><frame id="140582671781216" name="_main" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/main.py" line="296"></frame><frame id="94234068782320" name="wrap_session" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/main.py" line="240"></frame><frame id="140582904662944" name="pytest_cmdline_main" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/main.py" line="289"></frame><frame id="94234090491968" name="_multicall" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/callers.py" line="187"></frame><frame id="140582904656400" name="&lt;lambda&gt;" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/manager.py" line="84"></frame><frame id="94234090684240" name="_hookexec" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/manager.py" line="93"></frame><frame id="94234090554848" name="__call__" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/pluggy/hooks.py" line="286"></frame><frame id="94234067792464" name="main" file="/home/mowglie/.pyenv/versions/py3/lib/python3.8/site-packages/_pytest/config/__init__.py" line="157"></frame><frame id="140582986219072" name="&lt;module&gt;" file="/home/mowglie/.pycharm-2020.2.2/plugins/python/helpers/pycharm/_jb_pytest_runner.py" line="43"></frame></thread></xml>
debugger: received >>114 13 pid_12882_id_140583053184448 140582185093536 FRAME
<<
Received command: CMD_GET_FRAME 114 13 pid_12882_id_140583053184448 140582185093536 FRAME
debugger: received >>114 15 pid_12882_id_140583053184448 140582185093536 FRAME
<<
Received command: CMD_GET_FRAME 114 15 pid_12882_id_140583053184448 140582185093536 FRAME
print(t)
debugger: received >>121 17 pid_12882_id_140583053184448 140582185093536 FRAME print(t)@LINE@
<<
Received command: CMD_CONSOLE_EXEC 121 17 pid_12882_id_140583053184448 140582185093536 FRAME print(t)@LINE@

 

 

0
Comment actions Permalink

Note that it must have something to do with my python environment, because running the same test with a much simpler pyenv environment (with just python 3.8 and pytest) works fine...

0
Comment actions Permalink

By comparing the two logs (with simple env and more complex env), one of the main differences in the log is that for the complex env (where I have numpy, matplotlib, etc, installed), the log contains a section about `=== Scientific View debug enabled ===` which is not there for the simple one. Could that be the reason?

0
Comment actions Permalink

Fabien Maussion

Have you tried recreating the problematic interpreter and environment?

0
Comment actions Permalink

Andrey Resler thanks - fater some tries I have been able to isolate the pytest plugin which would cause the error. Now I have to understand what exactly in this plugin is causing problems in pycharm - I wonder if it's something with multiprocessing... I will see if I can test more.

0
Comment actions Permalink

Andrey Resler sorry for letting lingering this for so long. The problem persists: the debugger console or variable display hangs when at the breakpoing, although the tests are running fine without debugger. I can't find an easy way to reproduce this, but it seems that certain things we are doing in our code is disturbing PyCharm.

Setting PYDEVD_USE_CYTHON=NO and PYDEVD_USE_FRAME_EVAL=NO does not help.

Here are the two full logs with debug mode on: https://gist.github.com/fmaussion/9406aebb561bb9aa5cf02ebc435b0432

The relevant parts in the logs happen AFTER the tests start, because as I said the problem occurs only if I run some code of ours. And when I run our code, PYCHARM takes new decisions, such as:

 

```

=== Scientific View debug enabled ===

```

Then, later:

 

```

("The debugger thread 'DispatchReader' is traced which may lead to debugging performance issues.",)

```

What could be the reason for these changes? Debugging this problem is slowly driving me crazy...

 

 

 

 

 

0
Comment actions Permalink

FWIW: going back to 2019.3.5 did NOT work for me, but going back to 2019.1.4 did. Will stick to that version for the time being...

0
Comment actions Permalink

Fabien Maussion Hi, 

Earlier you said that you were able to isolate the issue to Pytest plugin. Do you mean this plugin by any chance?: https://plugins.jetbrains.com/plugin/10242-pytest

Does the issue reproduce if you disable the plugin?

0
Comment actions Permalink

Andrey Resler No, I'm not using any plugin on the PyCharm side. I was using a custom pytest plugin (which I disabled now) and at least debugging now works in dummy projects. However, debugging on my project still does not work with  current Pycharm (but it does with Pycharm 2019.1.4 which I'm using at the moment).

As I said, it was very difficult for me to isolate the code section which would brake PyCharm's debugger: what I'm doing is importing a module, which at import does a number of things, such as setting up a logger, opening and closing files, importing many packages of scientific python... And after all these lines of code are executed, PyCharms debugger is confused as visible in the logs I provided...

0

Please sign in to leave a comment.