PyCharm can't see python binaries even if they are there!

Managing and using multiple Python versions on the same computer can be... interesting. Before you look at PyCharm you should check what your system OS, Linux Mint, thinks it has for versions of Python.

It is made more tricky because most (Linux) distros will have some system installation of Python. Which may be any 3.x version, heck on the latest version of macOS it is still 2.7.

You need to systematically look at all the versions of Python on the computer. I would recommend the following (from a command prompt, not mc) as root:

1. run # which python and make a note of what it returns (path + file name + whether it is an alias). It may (as on my CentOS machine) say there is no python (as in Python 2).
2. let's assume it says "no python"
3. run # which python3 (just python3, no python 3 version) make a note of the directory, let say it returns '/usr/bin/python3' so directory is /usr/bin/
4. run # ls -la /usr/bin/python3 the whole path to the file does it point to an alias?
5. run # /usr/bin/python3 --version what is returned
6. You say your other version is 3.7?
   run # which python3.7 what directory does it say it is in? Let's say /usr/local/bin 
7. run # /usr/local/bin/python* what do you see?
8. run # /usr/local/bin/python3.7 --version

How did you install the 'non system', i.e. user, version of Python 3.7? When you installed it (if you installed it) did you install it as a user version completely separate from the system version of Python (3)? If you installed Python 3.7 from the OS package manager that can be a problem because that will change/affect the system installation of Python. My understanding (and I could be wrong) is that to install a 'user' version of Python separate from the Linux OS (so not using the package manager) you have to install (compile) Python from source.

Thank you for your answer.

As I said, python 3.6 is the system installation of python 3, and it was installed using the OS package manager.

I also found python3.7 in /usr/lib, (LIB not BIN) but I don't know who place it there, surely not apt-get. The system doesn't know anything about this and it is not obviously looking into /usr/lib for binaries ($PATH is set in the usual way, looking into /usr/bin, /usr/local/bin, etc), so "which python3.7" returns nothing.

I vaguely remember that python 3.7 came along with pycharm, but I don't remember exactly how. THe strangest thing, is that I have no clue where python 3.7 is. Surely it is NOT in /usr/bin, however PyCharm believes that it is right there!

PyCharm does not install Python, it uses the Python(s) it finds in the path. PyCharm definitely does not go installing Python versions by itself. PyCharm has config settings which may (someone from JetBrains would need to comment/confirm) store paths, but I don't think it would go overriding OS path settings.

If you found 'python3.7' (a directory, not a file, I take it) which is not the system Python version, in /usr/lib then, sorry to say, your machine's Python installations have been messed up. That is, a user installation of Python has gone into system (/usr/lib) directories. It will need to be cleaned up.

From a command prompt type "echo $PATH', make a note of it. Run PyCharm, from the PyCharm terminal window run 'echo $PATH', make a note of it and compare the two. The path show in PyCharm will very likely, definitely if you are using Python virtual environments, be different.

Hunt through the two sets of paths for Python versions. But again, if your machine has python3.7 (and python3.6?) in /usr/lib then your machine setup is messed up.

As regards the /usr/lib directory and versions of Python, on my CentOS Python application server I have:

# ls -la /usr/lib
drwxr-xr-x. 3 root root 27 Apr 16 03:38 python3.6.  (operating system version of Python)

# ls -la /usr/local/lib
drwxr-xr-x. 35 root root 8192 Jun 29 08:14 python3.8.  (user [me] installed version of Python, compiled from source)

Which is as it should be, this is also shown the same for /usr/bin and /usr/local/bin (system version = 3.6, user version = 3.8).

Once you have found the extraneous/incorrect version(s) of Python they likely will need to be removed.

Regarding installing.. if you go the Python.org download page (for example 3.8.3) 

https://www.python.org/downloads/release/python-383/

you will not see any binary releases for Linux/UNIX. It is definitely a 'roll your own', i.e. compile from source, scenario. You would remember if you had compiled Python from source... it's not difficult, but it is not trivial.

Hi and thank you again. I'm not 100% sure that the paths in /usr/lib are wrong: multiple installations of python can be available and /usr/lib will just keep the packages any of these needs.

I followed your suggestion by looking at the PATH variable. The pycharm shell that opens up tells me that I have /app/bin and /usr/bin in its PATH. However, NO /app path exists in my computer filesystem, therefore we are in a virtual environment or container which I have no clue. Where are its related settings? Is it a venv or what?

BTW: other of my pycharm projects have a venv that I explicitly set in the project folder, and I can manage, installing the packages I need. But for this project, I wanted to use the "system" python, which now appears to be a virtual system which is out of my control.

I'm probably missing some information here.

Thanks

/usr/lib and /usr/bin are really system used folders (despite being called 'usr'). Yes there will be a /bin directory on your machine but really the directory /usr/bin is managed by the OS, if you install a package yourself then, most often, it will be installed in /usr/bin (for example Midnight Commander).

The precedence of folders (which can be seen by looking at the path) is:

/usr/local/bin

/usr/bin

/bin

Read the accepted answer to this question: https://askubuntu.com/questions/308045/differences-between-bin-sbin-usr-bin-usr-sbin-usr-local-bin-usr-local

"I'm not 100% sure that the paths in /usr/lib are wrong" if you have 'user' (or more than one) versions of Python in /usr/bin or  /usr/lib instead of (if Python 3.7 installed) /usr/local/bin or /usr/local/lib then that is problematic, and it shows that the user version(s) of Python were not installed using 'make altinstall' after the Python source was compiled. Again if you have both Python 3.6 and 3.7 in the same folder, and particularly if those folders are /usr/*,  it will cause problems, it will confuse software.

If Python is compiled and installed with 'make altinstall' the system version of Python will not be touched and the user version will go in /usr/local/bin.

You may find documents on the Internet mentioning that you can use the packaging system to add some repository which has later versions of Python than the usual OS supplied version — fine. But installing via the package manager will overwrite the system version of Python. If you read documents on the Internet which instruct on compiling Python and then using 'make install' instead of 'make altinstall' then they are wrong. Using 'make install' will overwrite the system Python.

But you have more than one version of Python on the machine, how the second one got on there you don't know.

You also mention now that there Python virtual environments on the machine. How you setup and use virtual environments in PyCharm you need to read the PyCharm documentation. You can use the system installed version (3.6) of Python for projects but you should still use a virtual environment (for installing Python packages). 

If you are running virtual environments here is a find command which will find all copies of 'python' on your system. In the virtual env. the python command will be aliased to the actual python executable used.

find / \( -type f -or -type l \) -name python -exec ls -la {} \;

N.B. BE EXTREMELY CAREFUL RUNNING THE ABOVE COMMAND BECAUSE THE RESULT OF THE FIND COMMAND IS 'PIPED' TO THE EXEC COMMAND WHICH WILL RUN THE COMMAND STRING AFTER IT ON EVERY FILE FOUND BY 'FIND'. IF THE 'EXEC' COMMAND YOU TYPE IN CHANGES FILES THEN THEY WILL INSTANTLY BE AFFECTED BY WHATEVER THE COMMAND IS. The command here is ls -la which lists files and shows if a file is a symbolic link to another file.

Read up on installing and using multiple versions of python, especially how to have them isolated from each other; python virtual environments and how to use them in PyCharm.

Good luck.

Same here. Two screenshots - one showing the actual files under /usr/bin starting with python*. And one showing the add system interpreter UI. For some reason there is mismatch between the files showing with "ls -l" and the one showing in the UI. There are even files in the UI that are not showing with "ls -l". I.e. the "python3.7m-config" is showing in the UI, but there is no such file. These are under Linux Mint 20. It is based on the Ubuntu 20.04.

And just a note - running /user/bin/python works fine and is starting up the Python interpreter. As additional note - virtual environments created for Python 2.7 are also not working. When added the Pycharm states that this looks like invalid SDK. But when used under the terminal - it works just fine.

Thank you Patrick, but as a 10-years Linux user I think I know the basics about the Linux filesystem.

The point is another one: PyCharm is pointing to an alternate filesystem that resembles mine (e.g. the home folder points to my home folder) but has different system directories, such as the /app/ one. If I explore this filesystem with the pycharm terminal I find, e.g. : /app/bin/pip

Looking into my filesystem from my terminal with find and grep I cannot find 'app/bin/pip' not even hidden inside any part of my filesystem.

Where is this filesystem and whose responsibility this is I don't know. Any pointer to PyCharm or Python documentation would be helpful.

Well Leodardo as a 40 year user of Unix, MS-DOS, Solaris, Windows, Windows NT, Linux, macOS I think I know a bit beyond the basics of Linux filesystems.

I keep making some basic points to you which I don't see are being followed.

Firstly - fundamentally a Linux filesystem will have (searched from left to right) the following directories in the path:

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/

so that user installed software will be found first.

Secondly you should not have a 'user' version of Python (3.7) installed in the same directory (/usr/bin) as the system installation of Python (3.6) — full stop. If you have both installed in the same directory fix it before you do anything else — there really is no other choice. You should fix that before continuing any discussion here.

Here is a link about setting up a user version of Python on Linux Mint (they also mention PyCharm)

https://blog.softhints.com/install-python-3-7-setup-pycharm-linux-mint/

note the article says "make altinstall" (as well, of course, that the user version of Python has to be compiled from source).

Regarding your point about the 'app' folder. A directory named 'app' is clearly not one of the standard Linux filesystem directories

https://www.howtogeek.com/117435/htg-explains-the-linux-directory-structure-explained/

https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

So if the OS did not create the folder 'app' then that only leaves... you. No, not PyCharm, you.

You say you can see the folders like '/app/bin/pip' in the PyCharm terminal, are you sure there is not an '~' in there? But then in one sentence you quote an absolute path (i.e. starts with '/') so /app/bin/pip inside PyCharm but then when you say you search for the directory in the OS command prompt you state a relative path — "from my terminal with find and grep I cannot find 'app/bin/pip'". That is there is no leading '/' — that makes a big difference.

Most likely you are using virtual environments, when you load a project and activate the virtual environment (well actually not you, PyCharm, because it will automatically activate a virtual environment it finds when it loads a project) the prompt will change to have the venv inside brackets (some-app).

And as regards the comments from Funnybrum - your example is different, you are talking about Python 2.7 and Python 3.8. I believe current versions of Mint install Python 3.x as the system default, older versions installed Python 2.7. 

https://tecrobust.com/no-python-2-on-upcoming-ubuntu-20-04-lts-version/

So if you are running Linux Mint 20 then either you upgraded and Python 2 got left on your machine by the upgrade (not good) and also installed 3.x, or you installed Linux Mint 20 fresh and installed Python 2.7. Regardless, again, running two major different versions of Python in /usr/bin as you are is only going to cause you problems.

"running /user/bin/python works fine and is starting up the Python interpreter" — yes, the Python 2.7 interpreter, if that is what you want then fine, your choice. If you run python3 --version then you should see Python 3.8. As to what you are seeing in the PyCharm UI when you are choosing an interpreter and particularly that you see 3.7 and pyvenv which was deprecated in Python 3.6. 

https://docs.python.org/dev/whatsnew/3.6.html#id8

well my PyCharm is not confused in the same dialog box I see simply what is in the OS directory. The small errors pointing to top-right mean the file is a link. Talk to JetBrains.

Yep, this is Mint 20 with additionally installed Python 2.7. Using update-alternatives I've set the 2.7 version as default to see if this will help PyCharm to see the available system distribution. Obviously it didn't.

And the question for my case is pretty clear - why there are differences between the file system and the UI for choosing Python interpreter?

As side note - installing Anaconda2 managed to get the 2.7 interpreter available in PyCharm.

Dear Patrick, 

I'm glad your 40 years experience helped you guess after paragraphs of brag that I am having a problem of virtual environments, which I suggested from the beginning.

In other projects I have (venv) as shell, as I explicitly selected to use a virtualenv using python3.7. This is contained in the venv folder of the project and I have full control over it. With this folder, the terminal is just a 'sh' shell, not a venv, as I selected the "system interpreter". However PyCharm worngly takes it as python3.7 because *CLEARLY* PyCharm is not looking in the real filesystem but in a virtual environment which it made up, and I can't find how/where.

Once again, if there is a PyCharm dev who can tell if this is a bug, of if there are suggestions about virtual environments and/or PyCharm I'm glad to take them, and read the docs. However, I think I can skip the "Foundations of Operating Systems" class for today. Thanks for the effort.

@Leonardo

I'm not sure this is a bug, but it looks worth investigating. Could you please submit an issue to https://youtrack.jetbrains.com/issues and provide:

- Screenshots of the file listing in PyCharm

- Full output of 'ls -la /usr/bin'

- Logs from **Help | Collect Logs and Diagnostic Data**

Leonardo

I totally feel your pain and confusion. I am also on Linux Mint 20, reinstalled from scratch about a month ago.

I first installed PyCharm-community via the regular applications manager GUI (the one that comes with Cinnamon), These are the specs shown in my system for this installation.

Everything was fine for a while, I managed to create virtual environments via virtualenv on the console and PyCharm had no trouble seeing them an executing the python interpreters from them.

One weird thing that I noticed though, was that funny business with some paths not being visible form the the Python console inside Pycharm. That is, if I did something like `import os; os.path.exists( some_absolute_path )`  for a file that I knew existed on my system I would get false, where as the same sequence of commands run in a python console outside of pycharm (run via a regular console) would yield true... Go figure...

Another weird thing is the funny issue with `/app` directory under which the pycharm's executable seems to be located, as reported by  `top` or `htop` for instance...This is weird as I there is no such absolute path in the system. I am 100% positive about that `sudo ls -la /` , unless there is something about linux filesystems that I don't understand. 

I noticed though, that there was and `app` directory   `/home/teo/.var/app`  which seemed to contain Pycham's stuff

Today I had to install  `snapd`. After that my virtual environments no longer worked... For some reason Pycharm was now unable to see the path to the interpreter that was definitely there on the system under `/home/teo/git/my-project/venv/bin/python` and which is  actually a link to `/usr/bin/python3.7`.  So I wasn't able to even start a console. Pycharm was able to see other binaries in the same directory, which weren't links... So I suspect there is some weirdness going on for links that go from whatever virtual filesystem flatpak executables create (?). I don't really understand flatpak / snap etc...

THE SOLUTION:

I decided to go old-school about my Pycharm installation, so I did the following.

1. Exported my Pycharm settings via the menu: File > Settings > Export Setings  to a zip file directly under my home directory.

2. Removed my Pycharm installation from the Applications Manager.

3. Downloaded the tar.gz from (here https://www.jetbrains.com/pycharm/) and followed the instruction for extracting the tar.gz archive as explained here: https://www.lifewire.com/how-to-install-the-pycharm-python-ide-in-linux-4091033

4. Started /opt/Pycharm-XXXX/bin/pycharm.sh

5. Provided my settings file from step 1. when asked. 

Now Pycharm has no trouble seeing my virtualenvs  and, incidentally, this seems to have sovled another issue I was having with commands executed directly on the console taking forever...

I no longer trust snap or flatpak

I have the same issue.

Using Pycharm 2020.2, and until recently was using Python 3.7 as my base interpreter.

Whenever i start a new project it's always in a virtual environment running its own interpreter. I recently upgraded the base interpreter to Python 3.8 and now when i return to one of my older projects Pycharm has no interpreter installed and does not recognise the 3.7 interpreter in the venv. I can click terminal from within pycharm, which opens the venv terminal, type python3 and the python 3.7 interpreter will run.

If i click Pycharm, Preferences, Project Interpreter and try and add an interpreter, navigate to the venv folder, it does not display the python 3.7 interpreter, so can't select it.

My echo from the venv terminal is:

/Users/jp/PycharmProjects/salesforce/venv/bin:/Users/jp/.pyenv/bin:/Users/jp/.serverless/bin:/usr/local/Cellar/python/3.7.7:/usr/bin:/bin:/usr/sbin:/sbin:/Users/jp/Library/Python/3.7/bin:/usr/local/Cellar/python/3.7.7/bin:/usr/local/bin:/usr/X11/bin:/usr/local/Cellar/bin/:/usr/local/:/usr/local/sbin:/usr/local/mysql/bin:/usr/local/Cellar/ruby/2.6.5/bin

The first item in the path is my venv environment.

Hope this helps find the issue.

Thanks

JP

Hi,

I have same issue. I installed the PyCharm using flatpak and it is very frustrating and confusing that you know that your system Python interpreter is e.g. Python 3.9, but you see in the dialog only Python 3.7. When you try to open some file in /usr/bin and the list of files is completely different, then it can you drive a crazy. I think that it should be communicated somehow in Open dialog window, that user is listening files inside flatpak container and not system files.

Due to this fact I consider PyCharm as not much useful, because I cannot install the latest Python interpreter inside flatpak container. I can use only workaround using some ssh server.

I'm having the same issue.  PopOS 20.10 (Ununtu 20.10).  I installed Pycharm through the pop store, which in turn used flatpak.

After Uninstalling Pycharm from flatpak, I then installed it manually, and it worked perfectly.

To everyone affected:

PyCharm stores information about the file system in system directory https://www.jetbrains.com/help/pycharm/directories-used-by-the-ide-to-store-settings-caches-plugins-and-logs.html#system-directory

Try to delete/rename this directory and restart PyCharm. Does it solve the issue?

I'm having the issue. Linux Mint 20.04, pycharm installed using the package manager.

Andrey Resler all my PyCharm installation is under ~/.var/appcom.jetbrains.PyCharm-Professional/ with 3 folders: cache, config, data. I tried to delete the "cache" folder but nothing changed.

I'm going to delete the flatpak app and use previous post solution

Well, I also got this issue when using Opensuse Leap 15.2 and install Pycharm using flatpak.

Looks like the culprit is the flatpak. As soon as I install Pycharm using the toolbox, Pycharm can read my system python default correctly.

Maybe there's a bug or something at the flatpak

One more datapoint - I also installed with flatpak and experienced the same issues.

I re-installed with snap and the issue is fixed.

Apparently Fedora 34 also uses Flatpak for installations through the system software manager, so thanks to everyone who suggested a manual installation! Thanks for nothing to PatrickS for wasting so much of everyone's time who had to read through so much useless nonsense at the start of this thread.

Guys, thanks for the advices! Linux Mint XFCE, installation via Flatpak causes problems with venv and python version.
Installed using the instructions, everything works fine! Snap and flatpack go to the trash heap.

In my case, installing with snap caused the problem.  I uninstalled then reinstalled using the installer downloaded from Jetbrains' website then it worked as expected.  

Hi, I have the same problem. Pycharm Community flatpak (2022.3) on PopOS 22.04. Thank you Mateini for the solution.