Trying to understand which environment is active when launching Jupyter from PyCharm pro

Hello, 

Jupyter server configuration allows to define an interpreter which should be activated. See the details here https://www.jetbrains.com/help/pycharm/configuring-jupyter-notebook.html#configure-server

Please let me know if you have any questions. 

Hi Antonina and thank you for your reply.  Yes, I saw that and my configuration is below.  As you can see, it is set to 'venv' which contains all the required modules BUT as mentioned, unless I instead change the conda environment (to one that contains the desired modules) then my imports fail.  Again my question is why does my environment seem to depend on my 'conda' settings instead of 'venv'.

Also I had sub-questions to understand the information that I see in the terminal:

1. what do the parameters in parentheses mean e.g (venv) (base) xxx@yyy % 
2. does the order matter?
3. why can I change the order and how does that impact Jupyter server?

Hello, 

>As you can see, it is set to 'venv' which contains all the required modules BUT as mentioned, unless I instead change the conda environment (to one that contains the desired modules) then my imports fail.

Please attach screenshots preseting: 

1. notebook with the resolution error

2. interpreter configuration which includes the packages installed (https://www.jetbrains.com/help/pycharm/installing-uninstalling-and-upgrading-packages.html#interpreter-settings)

1. what do the parameters in parentheses mean e.g (venv) (base) - it means you have 2 environments activated:  conda environment called "base" and virtual environment called env
2. does the order matter? - I believe no
3. why can I change the order and how does that impact Jupyter server? - it is not related to Jupyter support at all. Do you install packages through terminal? 

My workflow is as follows:  I have a project created in PyC pro.  In the project I have a notebook directory where I create notebooks.  The notebooks import modules they require and also modules that I have created in the project.  PyC docs indicate that if I launch a jupyter server from the terminal in the PyCharm pro IDE, then the environment used by that jupyter server will be 'venv'.  When I do that I get the following error in my notebook:

BUT 'sklearn' _is_ in my 'venv' (and yes 'sklearn' is the correct name to call for import of scikit-learn)

Have I chosen the correct interpreter?  Yes.

Now if I look at the terminal, I have:

Note the parentheses ie. two environments.  Now I close the jupyter server and activate a conda environment that has similar packages to my 'venv' so the cli now looks like this:

Note the order and names of the environments and also that now all imports work. 

So when I look at that I'm confused:  is the jupyter server now using the 'venv' or is it using the conda environment?   My expectation was this:  when I launch jupyter server from the terminal in the PyCharm IDE it would use the 'venv' regardless of what conda environments are already installed (because yes, I have already a bunch of existing conda environments from previous work that didn't use PyCharm but note that all the packages in 'venv' were installed from the UI in PyCharm Preferences)

FWIW these are my terminal settings.  It seems like it should activate the 'venv'

Hello, 

Thank you for the screenshots. The fact there are 2 environments activated is definitely not how the things should work. I will investigate it and report as a bug.  Please run "deactivate conda" prior to start of the Jupyter server. Will it do the trick? 

Here is a corresponding issue https://youtrack.jetbrains.com/issue/PY-49954, please follow it for updates. 

Unfortunately, 'deactivate conda' doesn't help.  Below is the screenshot of shutting down the running notebook.   After shutdown, two environments exist.  After 'deactivate conda', the environment returns to my conda 'base' environment.  If I try to 'deactivate conda' again to get rid of 'base', I get the deprecation warning.  Then when I relaunch Jupyter, I get the import error again because the environment is conda 'base'.  But thank you for your time and efforts.

Interesting follow on.  Before I shut down PyCharm I decided to look up the manual for conda deactivate in case I had done it wrong.  I hadn't but just for fun I ran 'conda deactivate' again.  Surprisingly, that gave me now _two_ conda 'base' environments (see double parentheses).  Curious I ran 'deactivate' again and it looks like all conda environments have disappeared.  However, launching jupyter still does not activate 'venv' and so the imports still fail.  HTH

Please do the following: 

1. Close PyCharm

2. Open terminal out of PyCharm and run: 

conda config

conda config --set auto_activate_base false

2. Open PyCharm's terminal

3. Do you see the expected venv activated? Please run

which python

and provide me with the output. 

Run:

python 

import sklearn

Is it successful? 

Kind of successful!  The conda 'base' no longer appears activated (confirmed by opening new terminal). and  'venv' seems to be the active environment.  Importing 'sklearn' seems fine but  'jupyter notebook' does not launch even though both packages are included in 'venv' and the correct server is selected for jupyter.

Please make sure the "jupyter" package is installed. Run in the terminal: 

pip install jupyter

The package for some reason is not in the list. 

Actually, jupyter is installed eg. if launch a terminal outside of pycharm it's available. 

If I do a 'pip list --local', which I thought provides a list of the 'venv' pip installed packages, 'jupyter' is present but most other packages that are shown for the interpreter under 'preferences' are not.

If I do 'pip list', all packages are present including Jupyter.

If I look under preferences Jupyter is in the installed list for the chosen interpreter.  I have uninstalled and reinstalled but that does not seem to make a difference.

I was looking at the interpreter and 'discovered' the option to view the interpreter's paths and below is what I see.  There are paths to miniconda, is that appropriate?  Is it because my 'base' interpreter is from  'Users/.../miniconda3/bin/python3'?

MikeB, 

as long as you use miniconda installation, yes it is ok. 

Please run in the teminal in PyCharm:

pip freeze
 

 and send me the output. 

accumulation-tree==0.6.2
appnope @ file:///opt/concourse/worker/volumes/live/5f13e5b3-5355-4541-5fc3-f08850c73cf9/volume/appnope_1606859448618/work
async-generator @ file:///home/ktietz/src/ci/async_generator_1611927993394/work
attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work
autopep8 @ file:///tmp/build/80754af9/autopep8_1592412889138/work
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
blanka-cli @ file:///Users/michaelbourassa/git/blanka_cli
bleach @ file:///tmp/build/80754af9/bleach_1612211392645/work
bokeh==2.3.1
brotlipy==0.7.0
certifi==2020.12.5
cffi @ file:///opt/concourse/worker/volumes/live/0ef369cc-6ba0-47e7-75da-208c6400381d/volume/cffi_1613246948181/work
chardet @ file:///opt/concourse/worker/volumes/live/c798b2ee-88b1-4341-6830-161a92c2399e/volume/chardet_1607706832595/work
click @ file:///home/linux1/recipes/ci/click_1610990599742/work
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1598884132938/work
clustergram==0.5.1
colorcet==2.0.6
conda==4.9.2
conda-package-handling @ file:///opt/concourse/worker/volumes/live/a7e34989-4c54-4cb6-4156-4e58ee270730/volume/conda-package-handling_1603018121300/work
cycler==0.10.0
cytoolz==0.11.0
dask @ file:///tmp/build/80754af9/dask-core_1592842333140/work
decorator==5.0.7
defusedxml==0.7.1
distributed @ file:///opt/concourse/worker/volumes/live/bd66aa48-5cf5-4b60-6ed4-f204fff153f6/volume/distributed_1605066538557/work
entrypoints==0.3
et-xmlfile==1.0.1
fastparquet @ file:///Users/runner/miniforge3/conda-bld/fastparquet_1589376895225/work
glob2==0.7
greenlet==1.0.0
HeapDict==1.0.1
holoviews==1.14.3
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
ipykernel @ file:///opt/concourse/worker/volumes/live/88f541d3-5a27-498f-7391-f2e50ca36560/volume/ipykernel_1596206680118/work/dist/ipykernel-5.3.4-py3-none-any.whl
ipython==7.26.0
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
ipywidgets==7.5.1
jedi==0.18.0
Jinja2 @ file:///tmp/build/80754af9/jinja2_1612213139570/work
joblib @ file:///tmp/build/80754af9/joblib_1613502643832/work
jsonschema @ file:///tmp/build/80754af9/jsonschema_1602607155483/work
jupyter==1.0.0
jupyter-client==6.1.12
jupyter-console==6.4.0
jupyter-contrib-core==0.3.3
jupyter-contrib-nbextensions==0.5.1
jupyter-core @ file:///opt/concourse/worker/volumes/live/c8df8dce-dbb3-46e7-649c-adf4ed2dd00a/volume/jupyter_core_1612213293829/work
jupyter-highlight-selected-word==0.2.0
jupyter-latex-envs==1.4.4
jupyter-nbextensions-configurator==0.4.1
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
kiwisolver @ file:///opt/concourse/worker/volumes/live/0b2f3e77-eaa3-4995-7dd0-c994762fcbde/volume/kiwisolver_1612282417472/work
locket==0.2.1
Markdown==3.3.4
MarkupSafe @ file:///opt/concourse/worker/volumes/live/cb778296-98db-45ad-411e-6f726e102dc3/volume/markupsafe_1594371638608/work
matplotlib @ file:///opt/concourse/worker/volumes/live/41e8cd50-031f-4dda-5787-dd3c4f4e0f08/volume/matplotlib-suite_1613407855571/work
matplotlib-inline==0.1.2
memory-profiler @ file:///tmp/build/80754af9/memory_profiler_1593485047258/work
mistune @ file:///opt/concourse/worker/volumes/live/95802d64-d39c-491b-74ce-b9326880ca54/volume/mistune_1594373201816/work
mkl-fft==1.3.0
mkl-random==1.1.1
mkl-service==2.3.0
msgpack @ file:///opt/concourse/worker/volumes/live/d7400f3a-e5de-4e85-5d4c-0c984c648401/volume/msgpack-python_1612287157185/work
mypy==0.800
mypy-extensions==0.4.3
nbclient @ file:///tmp/build/80754af9/nbclient_1614364831625/work
nbconvert @ file:///opt/concourse/worker/volumes/live/2b9c1d93-d0fd-432f-7d93-66c93d81b614/volume/nbconvert_1601914875037/work
nbformat==5.1.3
nest-asyncio @ file:///tmp/build/80754af9/nest-asyncio_1613680548246/work
networkx @ file:///tmp/build/80754af9/networkx_1598376031484/work
notebook @ file:///Users/runner/miniforge3/conda-bld/notebook_1588887231072/work
numpy @ file:///opt/concourse/worker/volumes/live/5572694e-967a-4c0c-52cf-b53d43e72de9/volume/numpy_and_numpy_base_1603491881791/work
olefile==0.46
openpyxl==3.0.7
packaging @ file:///tmp/build/80754af9/packaging_1611952188834/work
pandas==1.0.3
pandocfilters @ file:///opt/concourse/worker/volumes/live/c330e404-216d-466b-5327-8ce8fe854d3a/volume/pandocfilters_1605120442288/work
panel==0.11.3
param==1.10.1
parso==0.8.2
partd==1.1.0
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pg8000==1.15.2
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow==8.2.0
plotly==4.8.1
prometheus-client==0.11.0
prompt-toolkit==3.0.19
psutil @ file:///opt/concourse/worker/volumes/live/0673cd4b-30c1-4470-7490-d8955610f5d5/volume/psutil_1612298002202/work
psycopg2 @ file:///opt/concourse/worker/volumes/live/d12e789f-5b6c-4947-7cc2-187f3a9c9696/volume/psycopg2_1612298599991/work
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pyarrow==0.15.1
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pyct==0.4.8
Pygments==2.9.0
pymongo==3.11.3
pyodbc==4.0.30
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
pyrsistent @ file:///opt/concourse/worker/volumes/live/ff11f3f0-615b-4508-471d-4d9f19fa6657/volume/pyrsistent_1600141727281/work
PySocks @ file:///opt/concourse/worker/volumes/live/85a5b906-0e08-41d9-6f59-084cee4e9492/volume/pysocks_1594394636991/work
python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/work
python-snappy @ file:///opt/concourse/worker/volumes/live/85e56842-5d01-472f-47d9-9a76fa267019/volume/python-snappy_1610133348440/work
pytz @ file:///tmp/build/80754af9/pytz_1612215392582/work
pyudorandom==1.0.0
pyviz-comms==2.0.1
PyWavelets @ file:///opt/concourse/worker/volumes/live/ea36e10f-66e8-43ae-511e-c4092764493f/volume/pywavelets_1601658378672/work
PyYAML==5.4.1
pyzmq==20.0.0
qtconsole==5.1.1
QtPy==1.9.0
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
retrying==1.3.3
ruamel.yaml==0.17.2
ruamel.yaml.clib==0.2.2
scikit-image==0.17.2
scikit-learn==0.24.2
scipy==1.6.3
scramp==1.1.1
seaborn==0.10.1
Send2Trash @ file:///tmp/build/80754af9/send2trash_1607525499227/work
six @ file:///opt/concourse/worker/volumes/live/5b31cb27-1e37-4ca5-6e9f-86246eb206d2/volume/six_1605205320872/work
sortedcontainers @ file:///tmp/build/80754af9/sortedcontainers_1606865132123/work
SQLAlchemy==1.4.11
tblib @ file:///tmp/build/80754af9/tblib_1597928476713/work
tdigest==0.5.2.2
terminado==0.10.1
testpath @ file:///home/ktietz/src/ci/testpath_1611930608132/work
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
thrift==0.11.0
tifffile==2020.10.1
toolz @ file:///home/linux1/recipes/ci/toolz_1610987900194/work
torch @ file:///opt/concourse/worker/volumes/live/f1ad46c0-b0e0-4a2c-42b1-36c80dc13116/volume/pytorch_1596288435440/work
torchvision==0.7.0
tornado @ file:///opt/concourse/worker/volumes/live/05341796-4198-4ded-4a9a-332fde3cdfd1/volume/tornado_1606942323372/work
tqdm==4.60.0
traitlets==4.3.3
typed-ast==1.4.2
typing-extensions @ file:///home/ktietz/src/ci_mi/typing_extensions_1612808209620/work
urllib3==1.26.4
wcwidth @ file:///tmp/build/80754af9/wcwidth_1593447189090/work
webencodings==0.5.1
widgetsnbextension==3.5.1
zict==2.0.0

I think I got the problem, sorry I should have noticed this before. 

The thing is that the virtual environment is configured  based on the conda python interpreter , this leads to the dependencies inconsistency. 

To be more clear, these steps has been performed https://www.jetbrains.com/help/pycharm/creating-virtual-environment.html , where as the correct way is https://www.jetbrains.com/help/pycharm/conda-support-creating-conda-virtual-environment.html .

Please try: 

1. Delete the existing interpreter configuration 

2. Delete the venv directory from the Project root

3. Create a new conda env as described https://www.jetbrains.com/help/pycharm/conda-support-creating-conda-virtual-environment.html

Let me know how it goes. 

Okay that seems to have done it.  I deleted as you recommended and followed the instructions in the link you sent.  Things are working as hoped.  Some observations:

1. the virtual environment is sensitive to the conda settings so your recommendation to change conda config so that the base environment is not automatically activated is important;
2. the construction of a conda environment is confusing.  When creating the new interpreter and loading packages there is a little green button that allows the user to select packages from conda repos or pypi repos.  It's not clear to me why that is an option if I'm creating  a conda environment.  It would make sense to me if the only option was conda repos, but as it was I had to expressly select the option which seems odd;
3. the instructions indicated I should get an option to create an 'environment.yml' file.  That didn't happen and it's not clear to me at the moment how to create one except by hand; and
4. my 'venv' directory is at 'Users/.../opt/miniconda3/envs/ .  I did not expect that.  Also, the 'venv' folder does not appear in the 'Project Files' pane of the IDE nor is it in the 'Preferences/Project:xxx/Project Structure' (I thought it would show as 'excluded' or something)

Anyways, thank you for the prompt assistance, it was much appreciated!

thank you for you feedback! 

Please vote for this ticket in order to increase its priority: https://youtrack.jetbrains.com/issue/PY-19974

>my 'venv' directory is at 'Users/.../opt/miniconda3/envs/ .  I did not expect that.  

The venv location can be specified during the interpreter configuration: