PyCharm Runserver command issues

Answered

Created January 26, 2017 21:26

I'm trying to make thumbnails from pdfs. To do it, I need to import wand.

It runs fine if I do `./manage.py runserver` from the command line, but if I run from PyCharm, it breaks.

When I step through the code, the problem is that the code used to open the blob is always empty. It's the right class (wand.image) but it's an empty one. The object I pass it is a pdf, but the blob conversion, which produces no error at all, is empty.

Again, if I launch the server from the command line, it works, but if I launch from PyCharm, it breaks.

Here's the code I'm running:

from wand.image import Image as WandImage
from wand.color import Color


def convert_to_thumb(pdf_path, slug):
    with open(pdf_path) as f:
        image_binary = f.read()
    all_pages = WandImage(blob=image_binary) #<-- Here image_binary is a pdf
    single_image = all_pages.sequence[0]     #<-- BOOM! all_pages is a wand.image, but it's empty. Gives an Index error
    with WandImage(single_image) as i:
        i.format = 'png'
        i.background_color = Color('white')  
        i.alpha_channel = 'remove'  
        i.transform(resize='x100')
        save_name = slug + '_pdf_preview.png'
        i.save(filename='/foo/bar/' + save_name)

        return i

19 comments

Ilya Kazakevich

Created January 27, 2017 19:49

Official comment

SysPath sometimes affects some libraries. E.g. library may need some external plugin and fail to find it due to incorrect sys.path.

You can also check environment variables, they may be different when tool launched from PyCharm and from terminal. That is all difference between PyCharm and terminal versions.

After it you may use debugger to debug WandImage internals.

Ilya Kazakevich

Created January 27, 2017 14:02

Hello.

Not sure what prevents WandImage from working (you probably need to debug it), but it could be syspath issue, since it is the only thing that may be different in PyCharm. Please try to evaluate "sys.path" when debugging or simply print it and compare output when launched from console and from PyCharm. It may give us a clue.

Rcladd

Created January 27, 2017 14:37

I actually doubt it's a syspath, since I would get an error when calling or even importing wand, right? There's no error at all until the attempt to sequence the empty wand.image object, which is just a standard Python error.

Here are the sys.path from each environment, the only difference I see is a couple are defined twice in PyCharm and the addition of PyDev.

PyCharm sys.path:

['/Users/rcladd/PycharmProjects/pycharm_myapp',
'/Applications/PyCharm.app/Contents/helpers/pydev',
'/Users/rcladd/PycharmProjects/pycharm_myapp',
'/Applications/PyCharm.app/Contents/helpers/pydev',
'/Users/rcladd/venv/myapp/lib/python27.zip',
'/Users/rcladd/venv/myapp/lib/python2.7',
'/Users/rcladd/venv/myapp/lib/python2.7/plat-darwin',
'/Users/rcladd/venv/myapp/lib/python2.7/plat-mac',
'/Users/rcladd/venv/myapp/lib/python2.7/plat-mac/lib-scriptpackages',
'/Users/rcladd/venv/myapp/lib/python2.7/lib-tk',
'/Users/rcladd/venv/myapp/lib/python2.7/lib-old',
'/Users/rcladd/venv/myapp/lib/python2.7/lib-dynload',
'/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/Users/rcladd/venv/myapp/lib/python2.7/site-packages',
'/srv/web/django_apps/myapp/myapp',
'/srv/web/django_apps/myapp']

CLI sys.path:



['/Users/rcladd/PycharmProjects/pycharm_myapp',
'/Users/rcladd/venv/myapp/lib/python27.zip',
'/Users/rcladd/venv/myapp/lib/python2.7',
'/Users/rcladd/venv/myapp/lib/python2.7/plat-darwin',
'/Users/rcladd/venv/myapp/lib/python2.7/plat-mac',
'/Users/rcladd/venv/myapp/lib/python2.7/plat-mac/lib-scriptpackages',
'/Users/rcladd/venv/myapp/lib/python2.7/lib-tk',
'/Users/rcladd/venv/myapp/lib/python2.7/lib-old',
'/Users/rcladd/venv/myapp/lib/python2.7/lib-dynload',
'/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/Users/rcladd/venv/myapp/lib/python2.7/site-packages',
'/srv/web/django_apps/myapp/myapp',
'/srv/web/django_apps/myapp']

Ilya Kazakevich

Created January 27, 2017 14:41

Can you run it as bare Python script in PyCharm instead of Django "run server" and see if it still does not work?

Rcladd

Created January 27, 2017 15:08

When I run it as a bare python script in PyCharm it fails with exactly the same error.

Traceback (most recent call last):
 File "<input>", line 1, in <module>
 File "/Users/rcladd/PycharmProjects/pycharm_myapp/publication/make_thumb.py", line 10, in convert_to_thumb
 single_image = all_pages.sequence[0] # Just work on first page
 File "/Users/rcladd/venv/myapp/lib/python2.7/site-packages/wand/sequence.py", line 112, in __getitem__
 index = self.validate_position(index)
 File "/Users/rcladd/venv/myapp/lib/python2.7/site-packages/wand/sequence.py", line 82, in validate_position
 'out of index: {0} (total: {1})'.format(index, length)
IndexError: out of index: 0 (total: 0)

When I run the same python command from the shell (same virtualenv) it runs fine.

Rcladd

Created January 27, 2017 15:47

Here's some more info:

When I run from CLI and use `pdb.set_trace()` to check the value of `all_pages` i get this

(Pdb) p all_pages
<wand.image.Image: 3da0549 'PDF' (612x792)>

But when I do the same thing from the PyCharm console, I get:

(Pdb) >? p all_pages
<wand.image.Image: (empty)>

Rcladd

Created January 27, 2017 20:00

@Ilya,

Thanks for your reply.

"SysPath sometimes affects some libraries. E.g. library may need some external plugin and fail to find it due to incorrect sys.path."

If the library were not found, it would give an error on import, yes? Regardless, the libraries are there when I debug the script.

"You can also check environment variables..."

They are the same in both cases.

"After it you may use debugger to debug WandImage internals."

Unfortunately, since wand is a wrapper for ImageMagick, there's no obvious way to step through the code.

What's clear is that PyCharm is somehow breaking the functionality of the script. Pretty frustrating.

Ilya Kazakevich

Created January 27, 2017 20:19

>If the library were not found, it would give an error on import, yes?

It may catch (using try / except) and suppress exception.

>They are the same in both cases.

os.environ are the same, including PATH? If you use native library, PATH could be an issue.

Rcladd

Created January 27, 2017 20:25

Hmm, I'll check for try/except in wand.

The PATH appears to be the same, as far as I can tell. I posted sys.path above.

In both cases, the ImageMagic library is located at /usr/local/lib/libMagickWand.dylib. I've verified this with PDB.

Ilya Kazakevich

Created January 30, 2017 18:06

You also can try to run it from terminal window in PyCharm to check it works. If not, it could be some kind of permissions issue

Rcladd

Created January 31, 2017 13:28

I've run from the PyCharm console, and it fails (see above)

When I try to open the terminal window from PyCharm, I get an error:

Can't Open Local Terminal
java.io.IOException: Exec_tty error:Unknown reason

Dmitry Trofimov

Created January 31, 2017 13:35

About the terminal: what do you have in the Settings | Terminal -> Shell path ?

Could you please attach the logs ( Help | Show logs..) ?

Rcladd

Created January 31, 2017 13:46

I don't have a Settings | Terminal.

But if I go to File | Default settings | Tools | Terminal I see that Shell path is pointing to a file that doesn't exist: /venv/app_name/bin/terminalactivate

What should it be?

Here are the logs for today:

2017-01-31 08:23:27,778 [404069822] INFO - .intellij.util.EnvironmentUtil - loading shell env: /bin/bash -c . '/Users/robladd/venv/xrdb3/bin/activate';'/Applications/PyCharm.app/Contents/bin/printenv.py' '/private/var/folders/p9/qkmlq50557sgbymmp_vzstv40000gn/T/intellij-shell-env.tmp'
2017-01-31 08:23:27,891 [404069935] INFO - .intellij.util.EnvironmentUtil - shell environment loaded (14 vars)
2017-01-31 08:23:36,031 [404078075] INFO - .intellij.util.EnvironmentUtil - loading shell env: /bin/bash -c . '/Users/robladd/venv/xrdb3/bin/activate';'/Applications/PyCharm.app/Contents/bin/printenv.py' '/private/var/folders/p9/qkmlq50557sgbymmp_vzstv40000gn/T/intellij-shell-env.tmp'
2017-01-31 08:23:36,141 [404078185] INFO - .intellij.util.EnvironmentUtil - shell environment loaded (14 vars)

Steven Klass

Created January 31, 2017 14:22

This does not appear to be a sys.path issue in that it's finding. I suspect multiple installs of wand. Try Intentionally inserting an Exception in line 112 of

/Users/rcladd/venv/myapp/lib/python2.7/site-packages/wand/sequence.py

Line 112:

   raise SystemError("Bummer")
   index = self.validate_position(index)

Then validate you can get this error when you run ./manage.py outside of PyCharm. If you don't sys.path has more than one version and that's the issue.

Rcladd

Created January 31, 2017 14:46

When I run from ./manage.py outside of PyCharm I get:

SystemError: Bummer

Steven Klass

Created January 31, 2017 15:36

Confirmed no sys.path issue. Something in PyCharm. Can you create a simple test case to prove it?

Rcladd

Created January 31, 2017 16:14

OK, I created a new PyCharm project `wand_test` using a virtualenv I created in PyCharm.

I did

pip install wand

from the command line (since I cannot open a terminal in PyCharm).

Then this file:

from wand.image import Image as WandImage
from wand.color import Color


def convert_to_thumb(pdf_file, filename):
    with open(pdf_file) as f:
        image_binary = f.read()
    all_pages = WandImage(blob=image_binary)
    single_image = all_pages.sequence[0]    
    with WandImage(single_image) as i:
        i.format = 'png'
        i.background_color = Color('white') 
        i.alpha_channel = 'remove'          
        i.transform(resize='x100')
        i.save(filename=filename + '.png')

And I ran it from the PyCharm console:

from make_pdf import convert_to_thumb
convert_to_thumb('diagnosis-of-strawberry-diseases.pdf', 'wand_test')

And got the (same) error:

Traceback (most recent call last):
 File "<input>", line 1, in <module>
 File "/Users/rcladd/PycharmProjects/wand_test/make_pdf.py", line 9, in convert_to_thumb
     single_image = all_pages.sequence[0]
 File "/Users/rcladd/venv/wand_test/lib/python2.7/site-packages/wand/sequence.py", line 112, in __getitem__
     index = self.validate_position(index)
 File "/Users/rcladd/venv/wand_test/lib/python2.7/site-packages/wand/sequence.py", line 82, in validate_position
    'out of index: {0} (total: {1})'.format(index, length)
IndexError: out of index: 0 (total: 0)

Steven Klass

Created January 31, 2017 16:16

I would post this to YouTrack as this is clearly a bug.

Rcladd

Created January 31, 2017 16:18

OK, I'll do that. Can I reference this support ticket in YouTrack?

Please sign in to leave a comment.