Anything with SSH Changed Recently, Specifically Agent Forwarding?

Previously I was able to run a python script on a remote server that connected to remote switch and did "things".   For some reason this is no longer working.  It appears the agent is not being forwarded to my remote server.  Specifically the "SSH_AUTH_SOCK" var is not being set during the execution of the remote ssh interpreter.   

For clarity:

[local-host] --> [remote-host] --> [network device]

I do develop of network scripting stuff on the "remote-host".  It has access to the "network device".   If I normally, from the command line, ssh into the "remote host" I can then ssh into the "network device".   I've been using pycharm to successfully develop in this environment for a long while.  I can "Run" my python scripts on the "remote host" through pycharm and they connect to the "network device" just fine.  Everything works great.

For some reason this has stopped working.  I notice that when I ssh from the cli into network device I can run "ssh-add -L" on the "remote host" and see all my identities.   However when done through pycharm's remote interpreter its not even connecting to the authentication agent.   If I try and run "ssh-add -L" using subprocess.Popen() python reports back that it cannot connect to the authentication agent.

If I print out os.environ I don't see the "SSH_AUTH_SOCK" being set.   

So, normal ssh works fine, through pycharm it does not.   I've tried this in two completely separate environments.  The only thing common is my local MacOS laptop and pycharm.   But with ssh working from my laptop and pycharm not...it seems this may be an issue with pycharm or its interpretation of some subsystem that I'm unable to trace down.

It's been a couple weeks since I worked on these tools so I don't know if Pycharm has been updated in that time.  It's largely likely that this isn't specifically a pycharm issue, but something else.  I need some help in figuring out what though since my dev workflow is now no longer working.

 

Happy to post any additional output, troubleshooting, logs, whatever, just let me know what's needed.

 

 

2
8 comments
Avatar
Permanently deleted user

Updates:

- Tried reverting to 2020.1.2 and 2020.1.1 and behavior did not change so not likely tied to an update.

- Tried a scorched earth approach:

Re-installed pycharm, setup remote server connection, using OpenSSH and Agent.   Still have the same issue.   So...I dunno

0
Avatar
Permanently deleted user

Turning on some debug stuff It looks like pycharm is picking up the right stuff:


2020-07-09 15:18:08,932 [ 11751] INFO - z.sshj.transport.TransportImpl - Client identity string: SSH-2.0-SSHJ_0.27.0
2020-07-09 15:18:08,947 [ 11766] INFO - z.sshj.transport.TransportImpl - Server identity string: SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
2020-07-09 15:18:09,087 [ 11906] INFO - om.intellij.ssh.impl.sshj.sshj - Authentication log: SSH connection to chip@netman-new:22
* With overridden host from OpenSSH config: 192.168.0.5
* Compression methods from config: none
* With signature factories: ssh-rsa, ssh-dss, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521, ssh-ed25519
* With keep alive interval 120 seconds
* With keep alive count max 3
* With OpenSSH-like host key verifier (knownHostsFile=[/Users/cgwyn/.ssh/known_hosts], strictHostKeyChecking=ASK, hashKnownHosts=false)
* With identity from OpenSSH config {/Users/cgwyn/.ssh/id_rsa.priv.home, without passphrase}
* With 2 identities from ssh-agent[/private/tmp/com.apple.launchd.J4lebD0VwD/Listeners]
* With authentications allowed by the client: publickey, publickey, publickey, password, keyboard-interactive
* Connected
=> none (failure, new auth methods allowed by the server: publickey, password)
=> publickey /Users/cgwyn/.ssh/id_rsa.priv.home (success)

However if I start a Python Console on the remote host (from within pycharm)

for k, v in os.environ.items():
... if 'SSH' in k:
... print("{} --> {}".format(k, v))
...
SSH_CONNECTION --> 192.168.0.240 56753 192.168.0.5 22
SSH_CLIENT --> 192.168.0.240 56753 22
SSH_TTY --> /dev/pts/1

There's no "SSH_AUTH_SOCK", not sure why it's not able to work with the agent to forward keys.

0
Avatar
Permanently deleted user

I have the exact same issue. Suddenly cannot debug code in a professional setting. Major blocker.

0
Avatar
Permanently deleted user

Any update on this? I am facing the same issue.

0

Hello, 

First of all, please vote for the corresponding feature request https://youtrack.jetbrains.com/issue/PY-45347 in order to increase its priority. 

As a possible workaround, please check the SSH_AUTH_SOCK in terminal and add it to Run/Debug Configuration environment variable section manually. 

Another workaround is to set the correct value for $SSH_AUTH_SOCK in a way so that it would be seen by all applications started from your Desktop session. For example, see this discussion: https://stackoverflow.com/questions/135688/setting-environment-variables-on-os-x

There is no workaround for the case where SSH_AUTH_SOCK changes dynamically, unfortunately.

 

 

0
Avatar
Permanently deleted user

I'm not sure the feature request captures the issue.

The issue appears to be PyCharm does not seem to honour ForwardAgent setting in SSH config.

Thus an available agent on my dev box is used to authenticate my SSH Interpreter, however the SSH agent is NOT passed to the SSH interpreter session - which is crucial.  Thus when I try to do a second SSH-hop from there it fails:

error authenticating: no auth sock variable

 

dev box [PyCharm] -> remote SSH interpreter -> 3rd box used by my python script

The problem is not the dynamic nature of SSH_AUTH_SOCK on the dev box, this should stay relatively static (in my case) - however Agent Forwarding dictates that a new auth sock is created on EACH run of the remote SSH interpreter *on the remote interpreter box*.  This is not happening in PyCharm despite my SSH config specifying "ForwardAgent yes".

If I do this from a regular linux console it works without issue.

To demonstrate the problem I can replace my remote interpreter with a bash script that echos out the value of the auth sock before executing python:

 

#!/usr/bin/env bash
echo "Auth Sock: *${SSH_AUTH_SOCK}*" >&2
python3 "$@"

 

 

PyCharm log gives us:

10:43 AM Couldn't refresh skeletons for remote interpreter
failed to run generator3/__main__.py for sshConfig://foo.bar@myserver:22 agent/home/foo.bar/pycharm-python, exit code 1, stderr:
-----
Auth Sock: **

 

 

As you can see no value is propagated.  Now if I run the same thing from a console window:

 

But from a terminal it's fine:

$ ssh myserver /home/foo.bar/pycharm-python
Auth Sock: */tmp/ssh-hcaJgwmSUH/agent.43465*

 

This value will be different with each run, by design.

 

I don't consider this a feature request - it's a bug - PyCharm SSH is obeying some (eg Host), but not all the config (eg ForwardAgent) of SSH Client Config, and these exceptions are not explained in the documentation.

 

 

 

0

Thank you for clarification. I do agree this issue should be treated as bug. Could you please submit a bug report to our issue tracker at https://youtrack.jetbrains.com/issues/py ?

0
Avatar
Permanently deleted user
0

Please sign in to leave a comment.