Running Scrapy + Conda configuration throws error but running in Anaconda terminal doesn't

I've set up Anaconda on my Windows 10 machine and I'm now trying to debug my Scrapy Spider with it.

Scrapy is installed on a virtual conda environment and running "(testscraping) C:\Projects\testscraping> scrapy crawl myspider" works perfectly well.

Now I want to be able to debug my script through PyCharm, but it keeps throwing errors at me and I'm having a hard time figuring out what I'm doing wrong.

 

I've set up my project interpreter to point to python.exe in the virtual env folder. However I can't see anywhere that this is correctly identified as a conda environment.

 

This works just fine and PyCharm recognizes the packages installed in the conda envrionment.

 

 

 

 

Then I've set up the configuration to execute "crawl myspider" through "cmdline.py" in my venv (also tried with the scrapy.cmdline as module instead with no luck)

 

Running this results in the following error:

 

However running the "scrapy crawl myspider" command through the Anaconda Prompt works perfectly fine.

 

 

Why does this happen? Is it because my Conda environment hasn't been activated when running the script? And if so how do I get PyCharm to activate it before running the script?

8 comments
Comment actions Permalink

Hi,

The only difference between conda prompt and your run configuration I see is that in the prompt you use scrapy executable, and in the run configuration - cmdline.py, which to be honest I have no idea how it works.

Try the following please:

1. Open terminal in PyCharm while in your project

2. "conda activate testscraping"

3. python c:\path\to\your\cmdline.py crawl myspider

Will you get the same error? If the answer is yes, this is most likely not an environment related issue. In that case maybe it's better to run scrapy from the terminal.

0
Comment actions Permalink

Thanks for the reply.

The thing is I'm using the Anaconda console to run the crawling, so I can't use the terminal from PyCharm, since "conda" isn't available from cmd or Powershell.

I have to use the cmdline.py script otherwise PyCharm would try to run something like "python scrapy.exe crawl myspider" which obviously wouldn't work.

0
Comment actions Permalink

Okay, so I've just tried to add the Anaconda3 dir to my PATH. This allowed me to run conda through the normal terminal. However activating my conda env in the terminal doesn't change anything. I'm still seing the error.

I've tried telling my "Run configuration" to call "conda activate testscraping" before running the actual script but that just results in the terminal throwing the following error: "CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'." even though I've run "conda init cmd.exe" in my terminal.

0
Comment actions Permalink

Seems like this is a known issue: https://youtrack.jetbrains.com/issue/PY-23417

I guess I'll just have to wait for 2019.1 to go out

0
Comment actions Permalink

Yes, 2019.1 will fix a numerous issues related to Anaconda, including this one.

0
Comment actions Permalink

Unfortunately, it has not been addressed in 2019.1 either. Come on, jetbrains. 

0
Comment actions Permalink

Ok, I've just tried it, and I don't think it has anything to do with Anaconda.

I was able to reproduce exactly the same issue from the terminal outside of IDE, just by using the `cmdline.py` script according to the above run configuration.

This can be worked around in PyCharm by using external tools. Something like below:

0
Comment actions Permalink

Checkout this answer ;)

0

Please sign in to leave a comment.