PyCharm remove python '-u' option with remote interpreter
Answered
I'm running my project via a remote pyspark interpreter. Everytime I run a script, PyCharm automatically adds the -u option to the run command.
For example, the full run command is `ssh://vito@7.7.7.7:22/usr/bin/spark-submit -u /path/to/script.py`.
Because I'm running the script with `spark-submit` I get an error as `spark-submit` does not have a `-u` option. I've tried editing the runconfig and removing the `PYTHONUNBUFFERED` default environment variable to no avail.
How can I stop PyCharm from adding the `-u` option to my remote interpreter's run commands?
Please sign in to leave a comment.
Hi Vcolano! Unfortunately, -u flag is hardcoded, meanwhile, you can try /usr/bin/python instead according to http://knowledge-shareing-xinyu.blogspot.ru/2016/02/how-to-programe-in-pyspark-on-pycharm.html instruction.
Thanks for the quick reply, Pavel.
I've taken a look at Xinyu's blog post but unfortunately it is only helpful if you want to run a spark job in local mode. To run a pyspark job in cluster mode (wherein spark distributes tasks to worker nodes, the reason to be using spark in the first place) you have to use the `$SPARK_HOME/bin/spark-submit` interpreter (or `$SPARK_HOME/bin/pyspark for spark < 2.0). The `spark-submit` command does not accept any options, so the `-u` immediately causes the command to throw an error hence preventing the ability to run pyspark jobs through a PyCharm remote interpreter.
Could you please inquire about adding an option to remove the `-u` flag? It seems rather inflexible that the flag is "hardcoded" in.
I see, thank you for the detailed response! I created a related ticket in our bug tracker: PY-28339. Please feel free to vote and comment!