pkg_resources in system install being used instead of virtual environment

On Windows, I am running a Python server that uses gino, asyncpg, and sqlalchemy. This application runs fine in normal Run mode, but when I run in debug mode, the database engine fails to initialize. I narrowed the issue down to a problem with a sqlalchemy script in which pkg_resources is used to load a dialect plugin for asyncpg. All of the other libraries in the project seem to be called from the virtual environment, but pkg_resources specifically is being loaded in from the system Python install:

Because my system install lacks the other dependencies, the loading process fails at this point.

But as you can see, pkg_resources seems to be installed just fine in my virtual environment:

When I try to import pkg_resources in the Python Console, it loads from the virtual environment. And when I run the application normally (not in debug mode), it seems to all work just fine.

Has anyone ever run into this issue? What in the world could be causing it? I am using PyCharm Community Edition 2019.3

9 comments
Comment actions Permalink

I would have to guess that debugger is using some other sys.path for some reason causing non-venv pkg_resources to take priority.

For starters, let's try adding print(sys.path) statement to your script, and compare results of running in normal vs debug mode. Do you see any non-venv paths printed?

Also, what kind of virtual environment are you using?

0
Comment actions Permalink

1. It looks like sys.path is essentially identical between Run and Debug.

2. I'm using a virtual environment installed with virtualenv, Python version 3.7.3. I've tried with different versions of Python.

Some additional information:

  • I'm running an app that injects uvicorn, which spins up another process. Turns out it's this second process where pkg_resources is imported improperly. 
  • During initial script run, sys.executable lists the python.exe in my virtual environment. After the spin-up of the uvicorn process, sys.executable lists the python.exe of my system.
  • My coworkers are using the same repo and their Debug mode works as expected.
0
Comment actions Permalink

I was able to get a minimal repro case. More details here: https://github.com/intrepidOlivia/bad-import-repro

0
Comment actions Permalink

Thank you for the repro case, but it also reproduces without debugger for me, even without using IDE at all, by running from the command line the executable is changed:


################################
------Repro Debug------
pkg_resources path: <module 'pkg_resources' from 'C:\\Home\\projects\\temp\\bad-import-repro\\venv_py37\\lib\\site-packages\\setuptools-40.8.0-py3.7.egg\\pkg_resources\\__init__.py'>
sys.executable: C:\Home\projects\temp\bad-import-repro\venv_py37\Scripts\python.exe
Asyncpg Dialects (Should not be empty): [EntryPoint.parse('postgresql.asyncpg = gino.dialects.asyncpg:AsyncpgDialect')]
-----------------------
INFO: Uvicorn running on http://0.0.0.0:9000 (Press CTRL+C to quit)
INFO: Started reloader process [6012]
------Repro Debug------
pkg_resources path: <module 'pkg_resources' from 'C:\\Home\\projects\\temp\\bad-import-repro\\venv_py37\\lib\\site-packages\\setuptools-40.8.0-py3.7.egg\\pkg_resources\\__init__.py'>
sys.executable: C:\Python\Python37\python.exe
Asyncpg Dialects (Should not be empty): [EntryPoint.parse('postgresql.asyncpg = gino.dialects.asyncpg:AsyncpgDialect')]
################################

So far looks like interpreter/environment issue.

0
Comment actions Permalink

It looks like it's not reproducing entirely - your Asyncpg Dialects list is not empty there. If you run it in Debug mode, is that list empty?

0
Comment actions Permalink

Also, which virtual environment utility and version are you using?

0
Comment actions Permalink

Update from my end: I uninstalled my virtualenv and re-installed it. Now I'm on version 16.7.8, and I can no longer reproduce this issue on the command line. I could yesterday, when I was using a different virtualenv, but I failed to document which virtualenv version I was using (I apologize).

I can still reproduce the issue when using PyCharm to set up a new interpreter. I can't find any info in the settings about what PyCharm uses to create its virtual environments.

What I would ask for is this: If you haven't updated your virtualenv yet, take note of the version. Then uninstall and reinstall virtualenv, and set up the virtual environment with the command line to see if the issue still reproduces.

0
Comment actions Permalink

>I can still reproduce the issue when using PyCharm to set up a new interpreter

This is what I initially did when I couldn't reproduce the issue.

Does the issue reproduce if you use the interpreter created in command-line in PyCharm? I mean if you add it as existing interpreter.

0
Comment actions Permalink

No, it seems that if I use virtualenv 16.7.8 to create the virtual environment outside of PyCharm, and then add it as an existing interpreter in PyCharm, everything works properly. So the issue seems to be with whatever PyCharm uses to set up its virtual environment?

0

Please sign in to leave a comment.