Debugging Calibre plugins using PyCharm
PyCharm Pro 3.1.1 Trial (Windows 7)
I thought I would try to see if it was possible to debug Calibre plugins
using PyCharm Pro since Calibre uses an embedded Python 2.7 interpreter. My understanding is that this should be possible by setting up a Python
Remote Debug configuration. I did that and then added the following to my plugin's __init__.py file:
When I debug my configuration I see in the Console window:
Starting debug server at port 12420
Use the following code to connect to the debugger:
import pydevd pydevd.settrace('localhost', port=12420, stdoutToServer=True, stderrToServer=True)
Waiting for connection...
And when I run the following in an external command window:
calibre-customize.exe -b .
to "install" my plugin using the source files from the current directory I get:
pydev debugger: CRITICAL WARNING: This version of python seems to be incorrectly compiled (internal generated filenames are not absolute)
pydev debugger: The debugger may still function, but it will work slower and may miss breakpoints.
pydev debugger: Related bug: http://bugs.python.org/issue1666807
pydev debugger: Unable to find real location for: threading.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_comm.py
pydev debugger: Unable to find real location for: calibre_plugins.count_pages.__init__
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_tracing.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\zipplugin.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\ui.py
pydev debugger: Unable to find real location for: site.py
At this point the calibre-customize program hangs and I have to Control-C to stop it. And calibre-customize is just installing the program. To actually run it you have to use calibre-debug. [Edit: I should try putting the settrace() call somewhere else.]
And the Debugger Console window then says:
Connected to pydev debugger (build 133.881)
pydev debugger: Unable to find real location for: threading.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_comm.py
pydev debugger: Unable to find real location for: calibre_plugins.count_pages.__init__
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_tracing.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\zipplugin.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\ui.py
pydev debugger: Unable to find real location for: site.py
I should mention at this point that Calibre dynamically loads plugins, so that's why it's looking in calibre_plugins.count_pages for my plugin. See here for details.
Okay, so I figured I had to set up some Path Mappings. In my case my "remote" machine is the same as the machine that is running PyCharm. Here's some of what I tried. If I map local:
<AbsolutePathTo>/Count Pages/__init__.py
to remote:
calibre_plugins.count_pages.__init__
and try again, I then see in the command window (omitting the same CRITICAL WARNING that never seems to change):
pydev debugger: Unable to find real location for: calibre_plugins.count_pages.__init__
pydev debugger: warning: trying to add breakpoint to file that does not exist: calibre_plugins.count_pages.__init__
(will have no effect)
pydev debugger: Unable to find real location for: threading.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_comm.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_tracing.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\zipplugin.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\ui.py
pydev debugger: Unable to find real location for: site.py
and the debugger console shows:
Connected to pydev debugger (build 133.881)
pydev debugger: Unable to find real location for: threading.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_comm.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_tracing.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\zipplugin.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\ui.py
pydev debugger: Unable to find real location for: site.py
So at least the Debugger Console window now stops complaining about __init__.py and the command window says something different than before.
The site-packages\calibre\ things it is complaining about are in my calibre.zip that I have added to my project interpreter's Paths.
I've also tried the same and other permutations for the other items with no success. Can someone tell me what Path Mappings I should be using?
I thought I would try to see if it was possible to debug Calibre plugins
using PyCharm Pro since Calibre uses an embedded Python 2.7 interpreter. My understanding is that this should be possible by setting up a Python
Remote Debug configuration. I did that and then added the following to my plugin's __init__.py file:
import sys
sys.path.append('../pycharm-debug-py3k.egg')
import pydevd
pydevd.settrace('localhost', port=12420, stdoutToServer=True, stderrToServer=True)
I have also set my project's Python Interpreter Paths to point at my calibre.zip I created to contain Calibre's python source files (its equivalent of a site-packages directory).When I debug my configuration I see in the Console window:
Starting debug server at port 12420
Use the following code to connect to the debugger:
import pydevd pydevd.settrace('localhost', port=12420, stdoutToServer=True, stderrToServer=True)
Waiting for connection...
And when I run the following in an external command window:
calibre-customize.exe -b .
to "install" my plugin using the source files from the current directory I get:
pydev debugger: CRITICAL WARNING: This version of python seems to be incorrectly compiled (internal generated filenames are not absolute)
pydev debugger: The debugger may still function, but it will work slower and may miss breakpoints.
pydev debugger: Related bug: http://bugs.python.org/issue1666807
pydev debugger: Unable to find real location for: threading.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_comm.py
pydev debugger: Unable to find real location for: calibre_plugins.count_pages.__init__
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_tracing.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\zipplugin.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\ui.py
pydev debugger: Unable to find real location for: site.py
At this point the calibre-customize program hangs and I have to Control-C to stop it. And calibre-customize is just installing the program. To actually run it you have to use calibre-debug. [Edit: I should try putting the settrace() call somewhere else.]
And the Debugger Console window then says:
Connected to pydev debugger (build 133.881)
pydev debugger: Unable to find real location for: threading.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_comm.py
pydev debugger: Unable to find real location for: calibre_plugins.count_pages.__init__
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_tracing.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\zipplugin.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\ui.py
pydev debugger: Unable to find real location for: site.py
I should mention at this point that Calibre dynamically loads plugins, so that's why it's looking in calibre_plugins.count_pages for my plugin. See here for details.
Okay, so I figured I had to set up some Path Mappings. In my case my "remote" machine is the same as the machine that is running PyCharm. Here's some of what I tried. If I map local:
<AbsolutePathTo>/Count Pages/__init__.py
to remote:
calibre_plugins.count_pages.__init__
and try again, I then see in the command window (omitting the same CRITICAL WARNING that never seems to change):
pydev debugger: Unable to find real location for: calibre_plugins.count_pages.__init__
pydev debugger: warning: trying to add breakpoint to file that does not exist: calibre_plugins.count_pages.__init__
(will have no effect)
pydev debugger: Unable to find real location for: threading.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_comm.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_tracing.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\zipplugin.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\ui.py
pydev debugger: Unable to find real location for: site.py
and the debugger console shows:
Connected to pydev debugger (build 133.881)
pydev debugger: Unable to find real location for: threading.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_comm.py
pydev debugger: Unable to find real location for: ..\pycharm-debug-py3k.egg\pydevd_tracing.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\zipplugin.py
pydev debugger: Unable to find real location for: site-packages\calibre\customize\ui.py
pydev debugger: Unable to find real location for: site.py
So at least the Debugger Console window now stops complaining about __init__.py and the command window says something different than before.
The site-packages\calibre\ things it is complaining about are in my calibre.zip that I have added to my project interpreter's Paths.
I've also tried the same and other permutations for the other items with no success. Can someone tell me what Path Mappings I should be using?
Please sign in to leave a comment.
One thing I've noticed, is that I have to stop & start the debugger to get it to notice changes in Path Mappings.
I also had my pydevd.settrace() in the wrong place. Instead of in my plugin's __init__.py, I should have placed it in actions.py, at a the place that will be called when the icon for my plugin is clicked.
Most importantly, I should have paid more attention to the message that is displayed in the PyCharm Editor (not the messages that are displayed in the debugger console or command window). That message says:
"calibre_plugins.count_pages.action can't be found in project.
You can continue debugging, but without the source."
So I click on the Edit Settings and add the following mapping
<AbsolutePathTo>/Count Pages/action.py
to:
Calibre_plugins.word_occurrences.action
After, I stop Calibre and restart the debugger then I indeed can step thru my plugin :)
A remaining fly in the ointment (at least on Windows 7), is that there doesn't seem to be any way to just specify a single path for all my plugin files. When I tried to use the following mapping:
<AbsolutePathTo>/Count Pages
to:
calibre_plugins.count_pages
(Note the lack of trailing / in the local path) I get the message:
"Remote file calibre_plugins.count_pages.action is mapped to
local path <AbsolutePathTo>\Count Pages.action and can't
be found. You can continue debugging, but without the source."
This seems to me to be a bug? PyCharm isn't able to tell that on my local system it needs to add a .py extension to properly map files in directories?
Another bug is if you have a trailing / in the local path. Then PyCharm converts that to a leading .? So the warning message becomes:
"Remote file calibre_plugins.count_pages.action is mapped to local
path <AbsolutePathTo>\Count Pages\.action and can't be found. You can
continue debugging, but without the source."
Things I haven't figured out yet but don't care so much about:
1) How to map modules that are in a local zip file? Do I instead have to
unpack it?
2) I am still getting a whole lot of "Unable to find real location for"
messages in the command window that I use to start calibre-debug (less
in the debugger console winow). A lot are for things in my calibre.zip
file but others are things in the standard python27\Lib
directory. Because of the apparent inability to correctly map paths to
directories (rather than single files), this would be painful to fix
if I was at all interested in stepping into standard library code.
answering your questions:
1) Paths to files inside zip like .../zipped_lib.zip/zipped_module.py should work.
2) There is an option to download sources of the file to show it in the editor - it is shown in the editor of the file with a missing content when you step in.
Does it help?