How to deal with "collecting data..." in debugger?

I have a function with a large array computation (about 100k objects). After its computation it's impossible to continue debugging due to terribly slow "collecting data..." operation in variables frame of debugger. It takes about 40-60 seconds after each debugger step to "collect data" and then it says "Unable to display frame variables" (however, sometimes it's evaluated eventually). Then if i click "show execution point" debugger will show me variables anyway. But if I click on "show execution point" before it says "Unable to display frame variables" it will not show me variables after appearing of error message even if i click again :(

I think the problem is connected with evaluating of text representation of my large array because array of the same size but without text representation of its objects doesn't slow debugger down.

I've googled that there should be an option that will allow to disable evaluating of text representation for specified types of objects but I can't find it :(
Source: https://www.jetbrains.com/idea/features/compiler.html#link28 (Configurable set of types to show as toString() by default, if I got it right)

I'm using PyCharm 4.0.5
15 comments
Comment actions Permalink
Official comment

Hi! We had this problem for a long time, but in PyCharm 2017.3 the option "Load Values Asynchronously" was implemented: https://youtrack.jetbrains.com/issue/PY-12987 (and it's turned on by default). 
If you still have the same problem with this option enabled, please create an issue in our issue tracker and provide steps for reproducing. Thank you!

Comment actions Permalink
By the way, it can be reproduced with following code:

class TestClass():
    field = 42

    def __repr__(self):
        return str([self.field ** .5] * 100)

def test():
    arr = []
    for i in range(300000):
        arr.append(TestClass())
    print('x')

test()

Put a breakpoint after cycle in test and you'll see
1
Comment actions Permalink

Still occurs, with both "Show return values" and "Show values inline toggled to off.

Debugger reaches breakpoint once. no data are readable.

Upon Resume Program (F8), debugger goes into an unknown state. Not paused, not taking requests.

2
Comment actions Permalink

Any updates on this issue? Unfortunately this makes PyCharm debugger useless for other than very simple and small programs.

There should be some option to disable detailed view for variables until specifically requested (view values of variables only if clicked). Or option to disable variable view completely, but still allow working with console (that's all I need).

For example the following code freezes debugger (remains in "collecting data..."). With N=100 or so everything still works. I have also enabled Gevent and disabled almost everything else with no help.

import numpy as np
N=10000
X={}
for i in range(0,N):
X[str(i)]=np.random.rand(200)

 PS. I'm running PyCharm 2017.2.3 with Python 3.6.2 on Win10.

 

 

 

1
Comment actions Permalink

I have the same problem. An official solution from the PyCharm team would be very appreciated!

0
Comment actions Permalink

The same problem in Idea Ultimate 2017.3.4 for Java debugger. Very frustratingly.

1
Comment actions Permalink

I have this problem as well on Ultimate 2017.2.5 for Java. Please consider optimizing it.

0
Comment actions Permalink

Do you have Memory view opened in debugger session? Try with it hidden if you have it opened.

Try also if it makes it better to disable toString() and alternative views in File | Settings | Build, Execution, Deployment | Debugger | Data Views | Java settings.

If it doesn't help please create new issue on YouTRack attaching idea.log ("Help | Show Log in..."): http://youtrack.jetbrains.com/issues/IDEA.

0
Comment actions Permalink

It has sped up quite a bit when toString() is disabled. Thanks for your help!

0
Comment actions Permalink

Any updated on this issue ? I am running Pycharm prof 2019.1 cannot debug , cannot add variables to watch

0
Comment actions Permalink

@Artyomk

If you tried different "Variables loading policies" and the issue is reproduced regardless, please submit an issue to https://youtrack.jetbrains.com/issues/PY , providing the detailed description and steps to reproduced. If you're already following a related issue, please post a link.

0
Comment actions Permalink

I had the same problem of variables not loading, then i found this answer in StackOverFlow, wherein it was mentioned to modify a configuration in Python's Debugger settings.

 

Select the checkbox 'Gevent Compatible'. This is in PyCharm. Probably this would work in Intellij Idea as well.

 

 

 

 

3
Comment actions Permalink

I solved the problem by:

2
Comment actions Permalink

I managed to break debugging in a way that looked almost exactly like this, but none of these solutions (presented here or elsewhere) worked. 

It was a relatively simple little object with only about 10-15 variables, but it locked every time with 'Collecting Data...'.  The weird thing was that it happened debugging under Visual Studio Code as well...  On trying to read the object both environments froze, and python3 leapt to the top of my process list with high CPU usage until I terminated the debugging session.

Ended up stepping through the code, as everything was working fine before this particular object was instantiated.  Discovered the innocuous line

version_number = payload_list[0] << 32 + payload_list[1]

which was trying to generate a 64-bit number out of two uint32_ts from a C embedded target.  Before this line, all worked fine - afterwards, nada.

Solution?  '+' binds tighter than '<<', so the code actually was doing this:

version_number = payload_list[0] << (32 + payload_list[1])

... And I was trying to debug an IMPLAUSIBLY big number. Presumably the debugger was freezing trying to represent that as a string for display.

So, if you're having this problem, particularly if the issue persists between environments, double check that you're not doing something daft, as I was.

1
Comment actions Permalink

PyCharm version 2020.03, solved by stonelazy's way. Thanks a lot.

0

Please sign in to leave a comment.