Debugging of subprocess.popen has unexpected results and fails

Hi People,

Can any of you please try the following code block and tell me if you get the same problem or if I'm doing something stupid as this is completely stopping me from debugging my scripts and I'm back to old school debug prints everywhere in my code!!

I have logged a bug for this here:
http://youtrack.jetbrains.com/issue/PY-7464

The following code which runs perfectly at the command prompt and PyCharm can also run it without issues however if I try debug it with PyCharm all hell breaks loose and directories are created based on my parameters even if I just run something simple like 'echo HelloWorld'

Code
#!/usr/bin/python3
import os
import subprocess
import sys

# Executes a command given an optional stdin byte string and returns a return code, stdout and stderr from the application
# Example:
# p4filename = "\"" + os.path.join(Variable.getvarvalue("P4PATH"), Variable.getvarvalue("P4FILE")) + "\""
# (retCode, stdout, stderr) = ExecuteCommand(p4filename + " set", None, mystring.encode('utf-8'))
def ExecuteCommand(cmd, cwd = None, stdin_bytes = None):
    if stdin_bytes != None and type(stdin_bytes) != bytes:
        print("ExecuteCommand stdin_bytes must be of type bytes.\n\tIf you have passed a string use your string.encode('UTF-8') to convert the string to a byte array of UFT-8 characters.")
        print("Type passed - %s" % type(stdin_bytes))
        sys.exit(1)
    stdout = ""
    stderr = ""

    if cwd is not None:
        cwd = os.path.realpath(cwd)
        if not os.path.exists(cwd):
            print( "***ERROR*** - Current working directory [%s] does not exist and therefore cannot execute command [%s]" % (cwd, cmd), {"status": "failed"})
            sys.exit(1)
    try:
        process = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, cwd=cwd)
        (stdout, stderr) = process.communicate(input=stdin_bytes)
        return (process.returncode, stdout.decode("cp437"), stderr.decode("cp437"))
    except OSError as ex:
        return (ex.errno, "", ex.strerror)

(retcode, stdout, stderr) = ExecuteCommand("echo HelloWorld")
print(retcode)
print(stdout)
print(stderr)

sys.exit(retcode)
prettyPrint();

Trying to debug the above even without breakpoints gives the following error:

============================================================
C:\Python32\python.exe "C:\Program Files (x86)\JetBrains\PyCharm 2.5.2\helpers\pydev\pydevd.py" –multiproc –client 127.0.0.1 –port 50488 –file D:/Scripts/test/Systray/Build/BuildBinProduct/test.py
pydev debugger: process 6832 is connecting
Connected to pydev debugger (build 117.663)
1

The system cannot find the path specified.
Error occurred while processing: .exe /c ECHO.
A subdirectory or file HELLOWORLD already exists.
Error occurred while processing: HELLOWORLD.


Process finished with exit code 1
============================================================

Any ideas as to what the problem is? I assume my code is not at fault because it works when you run it inside PyCharm or from the command line with expected results.


My system setup is as follows:

PyCharm 2.5.2
Python 3.2.2 (also tested with 3.0.1) from http://www.python.org/
Windows Vista x64
9 comments
Comment actions Permalink
This works as expected for me with PyCharm 2.6.
0
Comment actions Permalink
OK thanks for that Dmitry I'll upgrade to 2.6 and try again.
0
Comment actions Permalink
I've upgraded to Python 3.2.3 and PyCharm 2.6.1 but I'm still getting the same error:

Debugger Output
C:\Python32\python.exe "C:\Program Files (x86)\JetBrains\PyCharm 2.6.1\helpers\pydev\pydevd.py" --multiproc --client 127.0.0.1 --port 57293 --file D:/test.py
pydev debugger: process 5988 is connecting
Connected to pydev debugger (build 121.188)
1

The system cannot find the path specified.
Error occurred while processing: .exe /c ECHO.

prettyPrint();

If this is working for others then there has to be something different with my setup or configuration but I have no idea what to look for. Does anyone have any other ideas?
0
Comment actions Permalink
What is the working directory specified in the run configuration settings?
0
Comment actions Permalink
The working directory is D:\ and the script is set to d:\test.py

Do you have any pycharm packages installed for python? I don't have any but have seen some that come with the linux version of PyCharm installer and wondered if this could be a problem?

I currently have the following packages installed but this test.py script does not use any of them.

distribute    0.6.24    
p4python    2010.2.295040    
pip            1.0.2    
setuptools    0.6c11    
wsgiref        0.1.2    

Also I'm not using any virtual environments I'm directly using the Python 3.2.3 environment.
0
Comment actions Permalink
Does this work if you install PyCharm into a directory without spaces in the path?
0
Comment actions Permalink
A very good suggestion but sadly it does not appear to make difference.

C:\Python32\python.exe D:\PyCharm261\helpers\pydev\pydevd.py --multiproc --client 127.0.0.1 --port 58474 --file D:/test.py
pydev debugger: process 9444 is connecting
Connected to pydev debugger (build 121.188)
1

The system cannot find the path specified.
Error occurred while processing: .exe /c ECHO.
A subdirectory or file HELLO WORLD already exists.
Error occurred while processing: HELLO WORLD.


Process finished with exit code 0
prettyPrint();

Any other things I can try?
0
Comment actions Permalink
I have just created a new VMWare image of Windows XP, installed PyCharm 2.6.1 and python 3.0.1 and I still get this problem so  either there must be a setting in PyCharm that I am missing or this is a bug that everyone should be seeing with a fresh install as I cannot see from this test that it is an OS or environment issue although I guess we cannot rule it out completely.
0
Comment actions Permalink
I have now found and understand the problem which I believe is a bug in PyCharm and I also have a workaround for it listed here:

http://youtrack.jetbrains.com/issue/PY-7464

In short the problem occurs if the MULTIPROCESS option in the .idea\workspace.xml file is set to true. You have to manually set the back to false in the file and then things start working again. You cannot do this from the IDE because there appears to be another bug where the IDE (run/debug configurations window->Check no other instances are running) can set this to true but then cannot set it back to false again.
0

Please sign in to leave a comment.