[python] not on Plug-in Central ?

Hi,

It seems that the Python plug-in is not on the IntelliJ Plug-in Central,
so most users doesn't even know that IntelliJ could be used for Python development.

Thanks in advance,

Ahmed.

19 comments
Comment actions Permalink

Hello Ahmed,

AM> It seems that the Python plug-in is not on the IntelliJ Plug-in
AM> Central, so most users doesn't even know that IntelliJ could be used
AM> for Python development.

The Python plugin will be published in the Plugin Manager as soon as I'll
consider it good enough to be usable. Right now there are a few serious bugs
in it.

The work on the project is proceeding quite slowly, but it's not dead. :)

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

AM> It seems that the Python plug-in is not on the IntelliJ Plug-in
AM> Central, so most users doesn't even know that IntelliJ could be used
AM> for Python development.

The Python plugin will be published in the Plugin Manager as soon as
I'll consider it good enough to be usable. Right now there are a few
serious bugs in it.

The work on the project is proceeding quite slowly, but it's not dead. :)

Good to know that it's not killed :).
If I recall well, it was supposed to be published "immediately" after the release of 5.0 to
demonstrate the power of the Open Language API, since the JSPlug-in is using too many "internal"
things :).

IMHO it would have been even better if the "Custom Language Plugins.html" paper would have used the
Python plug-in as an example.
This would also bring more trust to API, since for now if we look
right (no mean to flame) there's no other plug-in that REALLY is using the complete Language API
except the JSPlug-in.

Ahmed.

0
Comment actions Permalink

It will be released by May 25.

Dmitry Jemerov (JetBrains) wrote:

Hello Ahmed,

AM> It seems that the Python plug-in is not on the IntelliJ Plug-in
AM> Central, so most users doesn't even know that IntelliJ could be used
AM> for Python development.

The Python plugin will be published in the Plugin Manager as soon as
I'll consider it good enough to be usable. Right now there are a few
serious bugs in it.

The work on the project is proceeding quite slowly, but it's not dead. :)

0
Comment actions Permalink

It will be released by May 25.

Thank you for the detail :).

I still have a few questions :) :
- Will it work with Demetra and/or with the 5.x version?
- What is the actual version (from the CVS on dev.java.net) really missing to be published
in the plug-in manager.
- When it is released, could it be used for real Python development, or it's just a
"proof-of-concept"? I mean, how complete is it in it's support for the typical working scenario?

AFAIK there are even pretty many scientific projects using Python, and most of them haven't even
heard of IntelliJ :).

Thanks in advance,

Ahmed.

0
Comment actions Permalink

Ahmed Mohombe wrote:
>> It will be released by May 25.

Thank you for the detail :).

I still have a few questions :) :
- Will it work with Demetra and/or with the 5.x version?


Right now we're shooting for 5.x only.

- What is the actual version (from the CVS on dev.java.net) really
missing to be published
in the plug-in manager.


Dmitry wants to finish the formatter, without it there are some
pains/annoyances with everyday editing, like having to frequently change
indent with tab/shift-tab.

- When it is released, could it be used for real Python development, or
it's just a "proof-of-concept"? I mean, how complete is it in it's
support for the typical working scenario?


It smells of proof of concept because:
- No interfile resolution (the resolution algorithm only knows about
definitions in the same file)
- No self.* resolution/completion
- No on-disk caches, files are reparsed on startup
- No Python module type
- No python stdlib configuration (like JDK selection for java projects)
- No run configurations
- No debugger
- No quick navigation with ctrln and ctrlshift+n

It has some more minor problems like formatter, odd structure view (due
to dynamic concept of fields in python).

It will not support real CPython debugging for at least a year due to
IDEA's lacking plugin API. I don't know if this will even go into IDEA 7
because JetBrains seems to want IDEA to stay mainly-Java IDE.

0
Comment actions Permalink

Thank you for the details Keith.

>>> It will be released by May 25.
>> Thank you for the detail :).
>>
>> I still have a few questions :) :
>> - Will it work with Demetra and/or with the 5.x version?


Right now we're shooting for 5.x only.

But if the changes in the API are so many (some said this on these lists), doesn't
it make sense to target Demetra, since if there's really API missing than only in
Demetra is possible to change this.

>> - What is the actual version (from the CVS on dev.java.net) really
>> missing to be published
>> in the plug-in manager.


Dmitry wants to finish the formatter, without it there are some
pains/annoyances with everyday editing, like having to frequently change
indent with tab/shift-tab.

IMHO it should release it earlier if it's possible, cause it might turn out that there are other
problems too.

>> - When it is released, could it be used for real Python development,
>> or it's just a "proof-of-concept"? I mean, how complete is it in it's
>> support for the typical working scenario?


It smells of proof of concept because:

:( :( :( :(.

- No interfile resolution (the resolution algorithm only knows about
definitions in the same file)
- No self.* resolution/completion
- No on-disk caches, files are reparsed on startup
- No Python module type
- No python stdlib configuration (like JDK selection for java projects)
- No run configurations
- No quick navigation with ctrln and ctrlshift+n

It has some more minor problems like formatter, odd structure view (due
to dynamic concept of fields in python).

:(. Most of the above would be required to use it for real development :(.

It will not support real CPython debugging for at least a year due to
IDEA's lacking plugin API. I don't know if this will even go into IDEA 7
because JetBrains seems to want IDEA to stay mainly-Java IDE.

Debugging would not be such a big problem for now since for Javascript there's also not debugging
support, but the rest, ... :(.

Ahmed.

0
Comment actions Permalink

Ahmed Mohombe wrote:
>> Right now we're shooting for 5.x only.

But if the changes in the API are so many (some said this on these
lists), doesn't
it make sense to target Demetra, since if there's really API missing
than only in
Demetra is possible to change this.


Hundreds of developers here use the python plugin and very few of them
plan to use demetra until 6.0 is released. There's no way to justify me
spending time porting to demetra.

>> Dmitry wants to finish the formatter, without it there are some
>> pains/annoyances with everyday editing, like having to frequently
>> change indent with tab/shift-tab.

IMHO it should release it earlier if it's possible, cause it might turn
out that there are other problems too.


The May 25 date is the result of this kind of thinking.

>> - No interfile resolution (the resolution algorithm only knows about
>> definitions in the same file)
>> - No self.* resolution/completion
>> - No on-disk caches, files are reparsed on startup
>> - No Python module type
>> - No python stdlib configuration (like JDK selection for java projects)
>> - No run configurations
>> - No quick navigation with ctrln and ctrlshift+n
>>
>> It has some more minor problems like formatter, odd structure view
>> (due to dynamic concept of fields in python).

:(. Most of the above would be required to use it for real development :(.


We work on it in spare time, and you could too. There is lots of
low-hanging fruit like ctrl+N support or run configurations, and
medium-hanging fruit like inter-file resolution. Even self.* resolution
wouldn't be difficult and would have huge payoff.

0
Comment actions Permalink

>>> Right now we're shooting for 5.x only.
>> But if the changes in the API are so many (some said this on these
>> lists), doesn't
>> it make sense to target Demetra, since if there's really API missing
>> than only in
>> Demetra is possible to change this.


Hundreds of developers here use the python plugin and very few of them
plan to use demetra until 6.0 is released. There's no way to justify me
spending time porting to demetra.

Well, I thought that this plug-in is developed by Jetbrains employees.
I use Demetra and 5.x in parallel. Mostly Demetra, and when something goes
wrong, I switch back to 5.1.1.

>>> Dmitry wants to finish the formatter, without it there are some
>>> pains/annoyances with everyday editing, like having to frequently
>>> change indent with tab/shift-tab.
>> IMHO it should release it earlier if it's possible, cause it might
>> turn out that there are other problems too.


The May 25 date is the result of this kind of thinking.

>>> - No interfile resolution (the resolution algorithm only knows about
>>> definitions in the same file)
>>> - No self.* resolution/completion
>>> - No on-disk caches, files are reparsed on startup
>>> - No Python module type
>>> - No python stdlib configuration (like JDK selection for java
>>> projects)
>>> - No run configurations
>>> - No quick navigation with ctrln and ctrlshift+n
>>>
>>> It has some more minor problems like formatter, odd structure view
>>> (due to dynamic concept of fields in python).
>> :(. Most of the above would be required to use it for real development
>> :(.


We work on it in spare time, and you could too.

No, I can't. If I could, I would have done it long time ago.
The very high complexity of the API, the total miss of documentation (you can't call those two html
documentation), totally overpasses my intellectual capabilities of being productive and being able
to do something useful with it.

Ahmed.

0
Comment actions Permalink

Hello Ahmed,

AM> The very high complexity of the API, the total miss of documentation
AM> (you can't call those two html documentation)

Sorry, but if you call this "miss of documentation", then I don't know
what you call "documentation".

The Language API for 5.0/5.1 has complete JavaDoc documentation for all relevant
APIs and a large overview/tutorial article describing how to use the API.
It's, in fact, the only area of the API that has such coverage.

Feedback like this doesn't exactly motivate us to produce better documentation
for other parts of the API - why bother if it's going to be called "totall
miss" anyway?

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Hello Ahmed,

>> - No interfile resolution (the resolution algorithm only knows about
>> definitions in the same file)
>> - No self.* resolution/completion
>> - No on-disk caches, files are reparsed on startup
>> - No Python module type
>> - No python stdlib configuration (like JDK selection for java
>> projects)
>> - No run configurations
>> - No quick navigation with ctrln and ctrlshift+n
>> It has some more minor problems like formatter, odd structure view
>> (due to dynamic concept of fields in python).
>>
AM> :(. Most of the above would be required to use it for real
AM> development :(.

Finding the hacks to fit Python's indentation model into the requirements
of the IDEA formatter core is not exactly a fun task, so not a large part
of my spare time is dedicated to it. I expect to be more productive when
I can finally start working on real features, like most of the ones in Keith's
list.

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

AM> The very high complexity of the API, the total miss of documentation
AM> (you can't call those two html documentation)

Sorry, but if you call this "miss of documentation", then I don't know
what you call "documentation".

Sorry, I did not expressed myself well(as always :) ). I wanted to express the "lack of documentation",
as the HTMLs supplied are(even if they exist) IMHO not of much help.
I don't know what's missing(I suppose just too much), but from those API documents, most of the
users who read them and I talked to (and I studied them too - several times to be precise), got more
confused than it clarified things.

I might be wrong, but as a proof of my theory is the fact that there's no other serious (and by
this I understand a complete and new language) implementation for that language API by others,
or even by Jetbrains (except that JS-plugin - but AFAIK that's using too much "internal" magic).
Another proof that it took so many years to publish a book. Even blatant and useless frameworks
got books in shorter time - one wonders why such a fantastic company didn't managed to
push some books (in all the domains - e.g. also plug-in development too) much earlier.

The Language API for 5.0/5.1 has complete JavaDoc documentation for all
relevant APIs and a large overview/tutorial article describing how to
use the API. It's, in fact, the only area of the API that has such
coverage.

I know you are brilliant in coding, as no other company was able to achieve what you've done.
This is fantastic, but regarding documentation - well, sorry - in this light of brilliant coding the
discrepancy with the documentation is even higher.
Maybe you should hire some specialists that are brilliant in writing docs and "communicating"
information, and not lose your time since you are the most productive in "doing" things not
"communicating" them to others :).

Feedback like this doesn't exactly motivate us to produce better
documentation for other parts of the API - why bother if it's going to
be called "totall miss" anyway?

Come one, this is no "kindergarden" :). You are a company, and you have a product and
want people to use it for as many things as possible, and as a target for
plug-ins too. So it is your interest to motivate users to write them, and not
wait for us to beg from you every line of documentation or explanation :).

Serious, if you read the forums, they're full with "please, please some more docs" :), - they're
even more than the donuts requests :).

In the hope that my strange style hasn't totally unmotivated you, I can't wait to see the first
"serious" implementation of that Language API :) (together with some docs :) ).

Thanks in advance,

Ahmed.

0
Comment actions Permalink

Hello Ahmed,

>> AM> The very high complexity of the API, the total miss of
>> documentation AM> (you can't call those two html documentation)
>>
>> Sorry, but if you call this "miss of documentation", then I don't
>> know what you call "documentation".
>>
AM> Sorry, I did not expressed myself well(as always :) ). I wanted to
AM> express the "lack of documentation",
AM>
AM> as the HTMLs supplied are(even if they exist) IMHO not of much help.
AM>
AM> I don't know what's missing(I suppose just too much), but from those
AM> API documents, most of the
AM> users who read them and I talked to (and I studied them too -
AM> several times to be precise), got more
AM> confused than it clarified things.
AM>
AM> I might be wrong, but as a proof of my theory is the fact that
AM> there's no other serious (and by
AM> this I understand a complete and new language) implementation for
AM> that language API by others,
AM> or even by Jetbrains (except that JS-plugin - but AFAIK that's using
AM> too much "internal" magic).

In fact, I really do not know what else can be done to improve the Language
API documentation. The complexity of the API is indeed high, no arguing with
that, but I've tried to explain it in the best way that I considered possible.

In my personal opinion, what really can help increase the Language API usage
is not better documentation, but rather the possibility to use existing parsers
and parser generator toolkits (ANTLR etc.). This task is on our roadmap,
but I'm not sure whether it'll make it into Demetra.

AM> Another proof that it took so many years to publish a book. Even
AM> blatant and useless frameworks
AM> got books in shorter time - one wonders why such a fantastic company
AM> didn't managed to
AM> push some books (in all the domains - e.g. also plug-in development
AM> too) much earlier.

This is a totally different issue. Organizing a book to be published has
little to do with API complexity, or people on the team.

AM> Maybe you should hire some specialists that are brilliant in writing
AM> docs and "communicating"
AM> information, and not lose your time since you are the most
AM> productive in "doing" things not
AM> "communicating" them to others :).

A writer for plugin documentation must be both a fairly good developer
(in order to be able to understand the APIs very well) and a good writer
(to be able to explain things well). As you can probably understand, such
people are quite hard to find.

>> Feedback like this doesn't exactly motivate us to produce better
>> documentation for other parts of the API - why bother if it's going
>> to be called "totall miss" anyway?
>>
AM> Come one, this is no "kindergarden" :). You are a company, and you
AM> have a product and
AM> want people to use it for as many things as possible, and as a
AM> target for
AM> plug-ins too.

Sure, but the company has some living people working on it, and the people
sometimes have feelings regarding work that they do and feedback that they
receive from users regarding their work. :)

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Hi Dmitry,

In my personal opinion, what really can help increase the Language API usage is not better
documentation, but rather the possibility to use existing parsers and parser generator toolkits
(ANTLR etc.). This task is on our roadmap, but I'm not sure whether it'll make it into Demetra.


Right, shame indeed. I still do not understand why JetBrains chose to standardise on JFlex instead of the more popular and full blown parser generators such as ANTLR or JavaCC. Beyond the lexical analysis, I have found the Language API to pretty much encourage developers to produce a hand-written parser which would be fine for the simplest of languages but harder to sustain, maintain and evolve for languages such as Groovy or Ruby.

So if not for Demetra, when can we hope to get even a first cut of the ANTLR integration?
It was first mentioned by Maxim Shafirov a year ago: http://www.intellij.net/forums/thread.jspa?messageID=3530209

Regards,

Franck

0
Comment actions Permalink

Hello Franck,

>> In my personal opinion, what really can help increase the Language
>> API usage is not better documentation, but rather the possibility to
>> use existing parsers and parser generator toolkits (ANTLR etc.). This
>> task is on our roadmap, but I'm not sure whether it'll make it into
>> Demetra.
>>
FR> Right, shame indeed. I still do not understand why JetBrains chose
FR> to standardise on JFlex instead of the more popular and full blown
FR> parser generators such as ANTLR or JavaCC.

The reason is simple: JFlex and hand-written parsers is what IDEA itself
uses internally, so we simply exposed the machinery we already have.

FR> Beyond the lexical
FR> analysis, I have found the Language API to pretty much encourage
FR> developers to produce a hand-written parser which would be fine for
FR> the simplest of languages but harder to sustain, maintain and evolve
FR> for languages such as Groovy or Ruby.

In my experience with Python, understanding how to write the parser was considerably
more difficult than actually writing it. The resulting parser code is quite
straightforward and works with very few problems. The only problematic part
was handling Python's indentation and putting whitespace in the AST tree
in places that the formatter could handle. I don't expect that updating the
parser for future Python version would be problematic.

FR> So if not for Demetra, when can we hope to get even a first cut of
FR> the ANTLR integration?
FR>
FR> It was first mentioned by Maxim Shafirov a year ago:
FR> http://www.intellij.net/forums/thread.jspa?messageID=3530209

No idea. The discussion of the post-Demetra plans is only just beginning,
and no details have been established.

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Have you ever tried writing an fault tolereant parser using an parser
generator? Some people state that there are many grammars already
defined using javacc. But these are not suited for an IDE but to parse
an (expected) valid file.
I think using jflex provides an good base because it's easy to implement
error recovery, reuse it for syntax highlihting and ast building.

Sven

Franck Rasolo schrieb:

Hi Dmitry,

>>In my personal opinion, what really can help increase the Language API usage is not better
>>documentation, but rather the possibility to use existing parsers and parser generator toolkits
>>(ANTLR etc.). This task is on our roadmap, but I'm not sure whether it'll make it into Demetra.


Right, shame indeed. I still do not understand why JetBrains chose to standardise on JFlex instead of the more popular and full blown parser generators such as ANTLR or JavaCC. Beyond the lexical analysis, I have found the Language API to pretty much encourage developers to produce a hand-written parser which would be fine for the simplest of languages but harder to sustain, maintain and evolve for languages such as Groovy or Ruby.

So if not for Demetra, when can we hope to get even a first cut of the ANTLR integration?
It was first mentioned by Maxim Shafirov a year ago: http://www.intellij.net/forums/thread.jspa?messageID=3530209

Regards,

Franck

0
Comment actions Permalink

Franck Rasolo wrote:

Hi Dmitry,

>> In my personal opinion, what really can help increase the Language API usage is not better
>> documentation, but rather the possibility to use existing parsers and parser generator toolkits
>> (ANTLR etc.). This task is on our roadmap, but I'm not sure whether it'll make it into Demetra.


Right, shame indeed. I still do not understand why JetBrains chose to standardise on JFlex instead of the more popular and full blown parser generators such as ANTLR or JavaCC. Beyond the lexical analysis, I have found the Language API to pretty much encourage developers to produce a hand-written parser which would be fine for the simplest of languages but harder to sustain, maintain and evolve for languages such as Groovy or Ruby.

So if not for Demetra, when can we hope to get even a first cut of the ANTLR integration?
It was first mentioned by Maxim Shafirov a year ago: http://www.intellij.net/forums/thread.jspa?messageID=3530209

Regards,

Franck


Yes, Maxim said last year that the problem is fault tolerance. You can
see comments to http://jetbrains.net/jira/browse/IDEA-4895, and of
course vote for the issue.

0
Comment actions Permalink

In fact, I really do not know what else can be done
to improve the Language
API documentation. The complexity of the API is
indeed high, no arguing with
that, but I've tried to explain it in the best way
that I considered possible.


I've written a custom language plugin to support the custom languages we use in our product and it has been a difficult process of experimentation and failure, figuring out how to implement various features. My impression of the custom language documentation is that it's merely an overview and that figuring out the details is left as an exercise for the reader.

Some examples of difficulty that I had:

- The exact mechanics of incremental lexing are unclear. For instance, there's no mention of how to cause relexing of incomplete input parsed as bad characters when the completing character is typed.

- Formatting model needs more explanation. I copied and pasted code from the Javascript plugin, not understanding some of the details.

For instance, when using FormattingModelProvider.createFormattingModelForPsiFile(), must the rootBLock parameter be the root block of the PsiFile provided or should it be the block that is being formatted, i.e. the block for the element requested in FormattingModelBuilder.createModel()?

How to implement FormattingBock needs more detail as well, in particular: getSubBlocks() and getSpacing(). Just pointing at the Javascript plugin's implementation is not enough. It took me days to figure out how it worked and how to get the result I wanted and I'm still not quite happy with it. I still have to figure out why my single-line comments insist on inserting a line-break before the next line. I think another problem I have is that I dislike the way the Javascript plugin suggests I have to implement getSubBlocks() and getSpacing(), with multiple if statements checking specific combinations of parent and child nodes to determine what indentation and spacing to use. It doesn't "smell" right.

- Implementing surrounders requires large amounts of what seems to be boiler-plate code, which I blindly copied and pasted from the Javascript plugin. It seems to me that the framework could be of some assistance in determining what elements lie between the startOffset and endOffset. It's literally a screenful of boiler-plate with one small for loop near the end that actually checks the elements themselves. Once again, very little explanation of all this work required is given in the documentation.

What I'm trying to get at is that while I don't expect it to be easy, I dislike having to guess so much.

Ciao,
Gordon

0
Comment actions Permalink

Given the uncertainties in software development, is May 25 (tomorrow!!) still
the planned release date for this plugin? :)

It will be released by May 25.



0
Comment actions Permalink

Given the uncertainties in software development, is May 25 (tomorrow!!)
still the planned release date for this plugin? :)

>> It will be released by May 25.
I wanted to ask that too :).

Ahmed.

0

Please sign in to leave a comment.