How to configure a Nix-based remote interpreter

Avatar
Permanently deleted user
Created

Hi,

my company workflow involves the use of remote servers and Nix. I would like to configure PyCharm in order to have the smoothest experience as if I were working locally.

A simple scenario would be:

  1. log in via SSH into the remote machine (possibly GCP).
  2. Enable the environment by launching a nix-shell.
  3. Edit and run Python code from there.

Mimicking this with PyCharm is not straightforward. In particular, I am able to set a remote Python interpreter, but by doing this everything is related to the host machine interpreter. Specifically, I can see two limitations:

  • The editor complains about missing packages and dependencies and therefore you cannot navigate through them (this is the most frustrating).
  • Running a script does not work because of these missing dependencies. Here, the workaround would be just deploying (without running) and next run it from a remote shell with nix activated.

Any help within this context is very appreciated.

 

Votes

1

Share

10 comments
Avatar
Andrey Resler
Created

Sorry, I'm not familiar with NIX, but I presume you're emulating virtualenv with nix-shell or something like this (https://nixos.wiki/wiki/Python) ?

Could you describe in more detail how do you manage your python environment on the remote host? 

0
Avatar
Permanently deleted user
Created

Basically, Nix has this clever way of "activating" the virtual environment by a heavy use of symlinks and env vars. Everything is cached in its internal store. These snapshots (profiles) can be called by invoking nix-shell. This action has the same effect of classic "source venv/bin/activate" command you usually do with virtualenv: everything lives in the "activated-shell".

It is like I need to connect to a remote "already-activated"-virtualenv shell. As far as I understood I think that in the end 3 general steps are needed (regardless the fact we use Nix or virtualenv):

  1. ssh connection to remote host (this is done properly)
  2. shell activation (I couldn't find anyhitng which let you inject something like this)
  3. launch python script inside the activated shell (usually this is done in the secure shell of step 1.)

Not sure I answered your question, I am not an expert.

0
Avatar
Andrey Resler
Created

Thank you. Which interpreter path have you specified when creating SSH interpreter? E.g. in case of normal venv it should be `venv/bin/python`

0
Avatar
Permanently deleted user
Created

I see where we are going. In fact, in that field I have the host Python path (/usr/bin) which is wrong.

However, pointing to the "correct python" does not seem to be enough. In fact:

  • ssh to the remote server
  • run nix-shell from the project root folder
  • `which python` -> "/nix/store/vhivkaa093jrfb0lfkhf2dl6qkzzhb70-python3-3.6.10/bin/python"
  • setting that path for the remote interpreter works but limited to the standard library, other pkgs are not visible.
0
Avatar
Andrey Resler
Created

Understood. It seems that it would require a special support for nix python environments, which we do not have (I doubt it would work if Nix was installed locally as well, but would be interesting to test this). I'm not aware of a workaround to make it work. Well, in theory you could create stubs for the packages which you want resolved, maybe there's a way to automate that, but I'm not sure. 

Meanwhile I created a feature request: https://youtrack.jetbrains.com/issue/PY-42461

Please leave a vote, feel free to comment.

0
Avatar
Permanently deleted user
Created

We are facing a similar issue: instead of nix-shell we install Python packages to the current profile via nix-env --install.

In case PyCharm is configured with a virtual env or system interpreter pointing to the wrapped Python interpreter script, everything seems to work fine. I tried a community edition, running on the remote host directly: 

However, in case SSH interpreter is set up, the python modules are visible on the interpreter setting only, auto complete does not work (look like packages are not indexed).

Using professional edition 2020.1, all modules are listed in project interpreter settings:

However, module not found, and no auto completion works:

0
Avatar
hacker1024
Created

I use this wrapper script as my Python interpreter to use a `shell.nix`. It slows things down a little, though, as `nix-shell` takes a couple of seconds to start.

#!/bin/sh
SCRIPTPATH="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )" 
PYTHONCMD="python $@" 
nix-shell "$SCRIPTPATH"/shell.nix --run "$PYTHONCMD"

 

2
Avatar
Nick M
Created

hacker1024 that wrapper script worked very well for me. Thank you for sharing!

Note: I had to name this shell file `python` in order for PyCharm to accept it as a valid Python interpreter.

The one thing I can't get working with this wrapper script is Coverage reports. I get the error message "Coverage is not importable in this environment" even though the Coverage package is installed.

0
Avatar
Aryeh Hillman
Created

Here is a script that I am using, which is similar to the above. It differs slightly in that it does not utilize `shell.nix`, but it does (a) solve a bug where packages cannot be found by IntelliJ (b) when using IntelliJ with Mac OS X. I may update it at some point to refefence a `shell.nix`.

https://gist.github.com/abhillman/1efa1d0498147b4dd0a9572e11abd373

0
Avatar
Aryeh Hillman
Created

>The one thing I can't get working with this wrapper script is Coverage reports. I get the error message "Coverage is not importable in this environment" even though the Coverage package is installed.

If you are using the script above from hacker1024, make sure that the coverage package is specified in `shell.nix` (i.e. not installed system-wide or by IntelliJ).

0

Please sign in to leave a comment.