Extend timeout delay on generator3.py

Answered

Hello,

 

I am currently struggling with the automatic timeout when generating skeletons on a new pycharm project (2018.3 and 2019.1).

 

I have my /home/ and and my project files on my local machine, but the codebase attached to my interpreter is huge (company code). Too large for generator3.py to return before it times out. I have once managed to get generator3.py to run in time, by setting up the interpreter without virtualenv and halting any other process on my host. But that is not a real solution as I need to work with a virtualenv.

 

As a result, I don't get any skeletons: Pycharm won't recognise "import sys" as a valid module, and won't unpack the PyQt4 libraries to map QtGui, QtCore,etc... 

 

As a temporary workaround, I copied the working python_stubs from the working interpreter into the python_stubs of my virtualenv - That 'kinda' works for auto-completion but it doesn't seem to take me to the original files when I ctrl-click on a PyQt4 object. Pycharm takes me to the cached skeleton file instead.

 

If I could somehow let generator3.py run five or ten more minutes, I'm fairly confident the skeleton generation would work as it should. Anyone has a clue where I could do that ?

 

Here is the error in idea.log :

# ------------------------------------------------------------

2019-03-14 10:34:36,662 [ 21988] INFO - ns.python.sdk.PythonSdkUpdater - Performing background update of skeletons for SDK Python 2.7 (venv_python) (/home/ps/all/workbench/.idea/venv_python/bin/python)
2019-03-14 10:34:36,664 [ 21990] INFO - .skeletons.PySkeletonRefresher - Refreshing skeletons for /home/ps/all/workbench/.idea/venv_python/bin/python
2019-03-14 10:34:36,674 [ 22000] INFO - .intellij.util.EnvironmentUtil - shell environment loaded (184 vars)
2019-03-14 10:35:52,472 [ 97798] INFO - mponents.impl.stores.StoreUtil - saveProjectsAndApp took 217 ms
2019-03-14 10:36:36,683 [ 142009] WARN - n.process.BaseOSProcessHandler - Process hasn't generated any output for a long time.
If it's a long-running mostly idle daemon process, consider overriding OSProcessHandler#readerOptions with 'BaseOutputReader.Options.forMostlySilentProcess()' to reduce CPU usage.
Command line: /home/ps/all/workbench/.idea/venv_python/bin/python /home/ps/all/local/pycharm-2019.1/helpers/generator3.py -L -s /home/ps/python:/home/ps/all/workbench/.idea/venv_python/lib64/python2.7:/home/ps/all/workbench/.idea/venv_python/lib64/python2.7/lib-dynload:/usr/lib64/python2.7:/usr/lib/python2.7:/usr/lib64/python2.7/lib-tk:/home/ps/all/workbench/.idea/venv_python/lib/python2.7/site-packages:/usr/lib64/python2.7/site-packages:/usr/lib64/python2.7/site-packages/gtk-2.0:/usr/lib/python2.7/site-packages: /home/ps/all/local/pycharm-2019.1/helpers/typeshed/stdlib/2 /home/ps/all/local/pycharm-2019.1/helpers/typeshed/stdlib/2and3: /home/ps/all/local/pycharm-2019.1/helpers/typeshed/third_party/2: /home/ps/all/local/ ...

java.lang.Throwable: Process creation:
at com.intellij.execution.process.BaseOSProcessHandler.<init>(BaseOSProcessHandler.java:33)
at com.intellij.execution.process.OSProcessHandler.<init>(OSProcessHandler.java:89)
at com.intellij.execution.process.OSProcessHandler.<init>(OSProcessHandler.java:44)
at com.intellij.execution.process.CapturingProcessHandler.<init>(CapturingProcessHandler.java:21)
at com.jetbrains.python.sdk.PySdkUtil.getProcessOutput(PySdkUtil.java:139)
at com.jetbrains.python.sdk.PySdkUtil.getProcessOutput(PySdkUtil.java:108)
at com.jetbrains.python.sdk.PySdkUtil.getProcessOutput(PySdkUtil.java:98)
at com.jetbrains.python.sdk.skeletons.PySkeletonGenerator.getProcessOutput(PySkeletonGenerator.java:197)
at com.jetbrains.python.sdk.skeletons.PySkeletonGenerator.listBinaries(PySkeletonGenerator.java:234)
at com.jetbrains.python.sdk.skeletons.PySkeletonRefresher.regenerateSkeletons(PySkeletonRefresher.java:300)
at com.jetbrains.python.sdk.skeletons.PySkeletonRefresher.refreshSkeletonsOfSdk(PySkeletonRefresher.java:124)
at com.jetbrains.python.sdk.PythonSdkUpdater$2.run(PythonSdkUpdater.java:185)
at com.intellij.openapi.progress.impl.CoreProgressManager$TaskRunnable.run(CoreProgressManager.java:731)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:164)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:586)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:532)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:86)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:151)
at com.intellij.openapi.progress.impl.CoreProgressManager$4.run(CoreProgressManager.java:403)
at com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:311)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2019-03-14 10:38:37,096 [ 262422] INFO - .skeletons.PySkeletonGenerator - Retrieving binary module list took 240416 ms
2019-03-14 10:38:37,097 [ 262423] ERROR - ns.python.sdk.PythonSdkUpdater - failed to run generator3.py for /home/ps/all/workbench/.idea/venv_python/bin/python: timed out.
com.jetbrains.python.sdk.InvalidSdkException: failed to run generator3.py for /home/ps/all/workbench/.idea/venv_python/bin/python: timed out.
at com.jetbrains.python.sdk.skeletons.PySkeletonGenerator.listBinaries(PySkeletonGenerator.java:254)
at com.jetbrains.python.sdk.skeletons.PySkeletonRefresher.regenerateSkeletons(PySkeletonRefresher.java:300)
at com.jetbrains.python.sdk.skeletons.PySkeletonRefresher.refreshSkeletonsOfSdk(PySkeletonRefresher.java:124)
at com.jetbrains.python.sdk.PythonSdkUpdater$2.run(PythonSdkUpdater.java:185)
at com.intellij.openapi.progress.impl.CoreProgressManager$TaskRunnable.run(CoreProgressManager.java:731)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:164)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:586)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:532)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:86)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:151)
at com.intellij.openapi.progress.impl.CoreProgressManager$4.run(CoreProgressManager.java:403)
at com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:311)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

 

 

6 comments
Comment actions Permalink

Here is the same log as a gitlab snippet, it's probably easier to read :

https://gitlab.com/snippets/1835420

0
Comment actions Permalink

Hi,

As far as I know there is no timeout for skeletons generation. How do you determine that the process is stopped?

Please upload your renamed idea.log to https://uploads.services.jetbrains.com/ after reproducing the problem and let me know the name of the file. We guarantee it's privacy.

1
Comment actions Permalink

Thank you for answering Sergey,

 

I have assumed there was a timeout based on this line :


2019-03-14 10:38:37,097 [ 262423] ERROR - ns.python.sdk.PythonSdkUpdater - failed to run generator3.py for /home/ps/all/workbench/.idea/venv_python/bin/python: timed out. 
com.jetbrains.python.sdk.InvalidSdkException: failed to run generator3.py for /home/ps/all/workbench/.idea/venv_python/bin/python: timed out.

I'd be interested to know what this log line refers to...

 

Unfortunately I can't give you a non-redacted idea.log, my company policy won't allow it. However I probably could go via the official channels for support since we have professional licenses - It's just that it will take more time with the timezone differences (oversea office) and the red tape.

 

I uploaded a copy of the failing log (python_venv) as well as a copy of the successful log (no venv)

 

The upload page said : 

Uploading...
File uploaded: idea_bug_report.log
Uploading...
File uploaded: idea_bug_report_no_venv.log
Upload complete!

 

Hopefully these can give a enough clues. I have seen other user having issues with PyQt4 and sys import , and I suspect this generator3.py problem might be the underlying issue.

 

0
Comment actions Permalink

Ok, I discussed it with our technical lead and it seems, looking at your log, that PyCharm times out on listing binaries.

I will not post the exact command PyCharm executes since it will expose some of your company data, but you can find it in your idea_bug_report.log by time "2019-03-14 10:36:36,683". What goes after Command line: ... is executed by PyCharm command to start listing binaries.

Since your attached company code base is huge (and we do not have access to it) we cannot dig deeper and tell you the exact reason why it times out. But you could run the above mentioned command outside of PyCharm and try troubleshooting it to see where and why it times out. generator3.py is written in Python (well, obviously) and you can understand what it does.

0
Comment actions Permalink

Hi & thanks for those details,

 

The company code is added in part of the PYTHONPATH, I could omit some paths but I actually prefer to have all included in the skeletons, rather than omitted. Some of that code is quite ancient and most of the "extra" time comes from modules that have heavy code at the module-level.

 

If I run the full CMD from the logs, it takes 4"40s to execute. Unfortunately Pycharm isn't as patient : (

 

So to be clear : generator3.py works fines, runs all the way to the end. But It seems like Pycharm can't wait long enough. Would there be any trick/hack to get that timeout to be a little more tolerant ?

Otherwise I could try to pre-run generator3.py and cache the result, to feed it back to Pycharm in less than the ~2mn timeout. I could even set that up as a routine job, but it's a bit of a stupid workaround if we can do anything to the timeout instead...

0
Comment actions Permalink

Unfortunately, you cannot change the timeout anyhow. It is hardcoded on java side.

0

Please sign in to leave a comment.