Cannot find pyspark library using Docker in python interpreter
I create a remote docker image and then I create a new interpreter in this way:
Add new interpreter → On Docker .. → Pull or Use existing
and everything is ok.
Unfortunately I notice that pyspark library is not shown as a package:

I try using the library through the docker image and everything works.
These are the installaed packages:
root@30a9f44932ed:/opt/bitnami/spark# pip list
DEPRECATION: Loading egg at /opt/bitnami/python/lib/python3.12/site-packages/pip-23.3.2-py3.12.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
Package Version
--------------- -----------
boto3 1.35.42
botocore 1.35.42
distlib 0.3.8
filelock 3.16.1
jmespath 1.0.1
numpy 2.1.2
pandas 2.2.3
pip 24.2
pip 23.3.2
platformdirs 4.3.6
py4j 0.10.9.7
pyspark 3.5.3
python-dateutil 2.9.0.post0
pytz 2024.2
s3transfer 0.10.3
setuptools 70.3.0
six 1.16.0
tzdata 2024.2
urllib3 2.2.3
How is it possible?
Can you help me?
Thanks
Please sign in to leave a comment.
Hi Basilico1994, I am wondering, which version is in use?
Have you tried File | Invalidate Caches | Invalidate and restart? Does it help by any way?
If not, would you please share logs from Help | Collect Logs and Diagnostic Data for in-depth analysis of the issue? Please upload it to https://uploads.services.jetbrains.com and let me know the upload ID.