Using PyCharm Jupyter with sparkmagic's PySpark kernel

Answered

I have installed sparkmagic in my Jupyter notebook environment.
I see sparkmagic's kernel PySpark appear in the kernel's dropdown box when I connect to Jupyter server (either managed or remote)
But when I attempt to execute a cell I get an error box with message "Can not execute non-Python notebook cells (notebook language is )."

I am trying to use Jupyter notebook to work with spark via Livy (this is what sparkmagic kernels are supposed to provide)

Any suggestions ?

4 comments
Comment actions Permalink


I managed to figure out how to resolve the issue. The problem is that kernel specification file for sparkmagic kernel pysparkkernel omits explicit "language" property:
https://github.com/jupyter-incubator/sparkmagic/blob/master/sparkmagic/sparkmagic/kernels/pysparkkernel/kernel.json
------
{"argv":["python","-m","sparkmagic.kernels.pysparkkernel.pysparkkernel", "-f", "{connection_file}"],
"display_name":"PySpark"
}
------

I fixed it by patching installed sparkmagic code in my venv:
emacs <my project root>/venv/lib/python3.7/site-packages/sparkmagic/kernels/pysparkkernel/kernel.json

It now looks like this:
---------
{"argv":["python","-m","sparkmagic.kernels.pysparkkernel.pysparkkernel", "-f", "{connection_file}"],
"display_name":"PySpark",
"language":"python"
}
--------

I then re-register the kernel (not sure if this was necessary, but I did it just in case) by running:
jupyter-kernelspec install sparkmagic/kernels/pysparkkernel

from my venv

After that both managed Jupyter and remote (when Jupyter is started outside of PyCharm) things started working as expected when "PySpark" kernel was used from the dropbox.

0
Comment actions Permalink

Hi,

Thanks for posting your workaround here.

We don't officially support non-Python notebooks at the moment.

Feel free to vote for https://youtrack.jetbrains.com/issue/PY-38331 and follow for updates.

1
Comment actions Permalink

Sergey,

Thank you for the clarification !

I am not sure what is your precise definition of “Python notebook”, but i am using Python notebook with sparkmagic pyspark kernel which is implemented in Python and accepts Python code.

It is unfortunate that sparkmagic implementers did not explicitly communicate language information in pyspark kernel metadata.

With remote Jupyter notebook one can not tell what language a specific kernel supports other than from the metadata supplied by the kernel.

What exactly are the constraints of what is considered supported by PyCharm Jupyter integration ?

0
Comment actions Permalink

I just discussed it with our developer. The problem is also on our side because we could actually support it out of the box.

I filed an issue about it https://youtrack.jetbrains.com/issue/PY-38576, please vote for it and follow for updates.

0

Please sign in to leave a comment.