Perl5 plugin for Intellij IDEA
Hi everyone.
Recently i've decided to try to make perl5 plugin for InterlliJ IDEA. I've seen feew attempts to start, but they've failed after creating four base classes :)
The problem with perl is his too free syntax, which requires a very custom lexer and parser. I've tried to port perl's original lexer, but it's too big, got lot of legacy stuff, so i've droped the idea.
Currently, i'm making lexer with JFlex and it works, but need tunings for different perlish situations. Anyway, some syntax is already highighted:
The second problem is that language plugins development documentation looks outdated (of course, not so many readers), but examples helps in such cases. Sometimes...
But is there some gurus in language plugins who could help with tricky questions or just to save some time in useless tries?
Also, if you would like to participate in perl5 plugin development, you are welcome: https://github.com/hurricup/Perl5-IDEA
Please sign in to leave a comment.
Yep, it seems indenting being done after it.
Sure I could, but it can be plugin developer bug, not IDEA itself. And not sure how to reproduce it in IntelliJ product.
Basically it's injection to multi-line PSI element (In JFlex Support it's Java injection):
If you hit enter after guessBareword(); cursor jumps to the beginning of Injection.
If indexing being done by word, not lexem, it doesn't matter than how words scanner's lexer recognizes Foo::Bar, right? Just should be in set of code tokens.
Well, WordOccurrence requires correct start and end offsets, so Foo and Bar should be within the file's text. Otherwise it doesn't matter how it works exactly.
Yep, seems it's plugins bad injection support, because CoffeScript plugin works fine with multiline injection. Now need to find out the problem
Got a problem and don't know how to handle it or even start to dig (even hard to explain):
Here is the parsed file:
... Some HTML hereIt's MultiplePsiFilesPerDocumentFileViewProvider with Perl as a base language and HTML as TemplateLanguage for TemplateLanguageFileViewProvider.
PHP-like markers are Perl comment tokens. Everything works fine until there is at least one symbol after last ?> token (newline white space, whatever). But if I put cursor there and click a backspace to delete this last symbol, getting an exception below. Lexing is fine but something bad happening outside of my plugin:
[ 10998] ERROR - .psi.impl.DocumentCommitThread - IdeaLoggingEvent[message=commitDocument left PSI inconsistent: HtmlFile:test.thtml; file len=5; doc len=7; doc.getText() == file.getText(): false, throwable=java.lang.Throwable
at com.intellij.psi.impl.DebugUtil.currentStackTrace(DebugUtil.java:497)
at com.intellij.idea.IdeaLogger.error(IdeaLogger.java:98)
at com.intellij.psi.impl.DocumentCommitProcessor.a(DocumentCommitProcessor.java:236)
at com.intellij.psi.impl.DocumentCommitProcessor.access$000(DocumentCommitProcessor.java:50)
at com.intellij.psi.impl.DocumentCommitProcessor$1.process(DocumentCommitProcessor.java:138)
at com.intellij.psi.impl.DocumentCommitProcessor$1.process(DocumentCommitProcessor.java:126)
at com.intellij.psi.impl.PsiDocumentManagerBase.finishCommitInWriteAction(PsiDocumentManagerBase.java:322)
at com.intellij.psi.impl.PsiDocumentManagerImpl.finishCommitInWriteAction(PsiDocumentManagerImpl.java:133)
at com.intellij.psi.impl.PsiDocumentManagerBase$3.run(PsiDocumentManagerBase.java:292)
at com.intellij.openapi.application.impl.ApplicationImpl.runWriteAction(ApplicationImpl.java:931)
at com.intellij.psi.impl.PsiDocumentManagerBase.finishCommit(PsiDocumentManagerBase.java:289)
at com.intellij.psi.impl.DocumentCommitThread$5.run(DocumentCommitThread.java:521)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733)
at java.awt.EventQueue.access$200(EventQueue.java:103)
at java.awt.EventQueue$3.run(EventQueue.java:694)
at java.awt.EventQueue$3.run(EventQueue.java:692)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:703)
at com.intellij.ide.IdeEventQueue.e(IdeEventQueue.java:734)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:569)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:382)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
]
[ 11005] ERROR - .psi.impl.DocumentCommitThread - PSI is broken beyond repair in: HtmlFile:test.thtml
java.lang.Throwable
at com.intellij.openapi.diagnostic.Logger.error(Logger.java:115)
at com.intellij.psi.impl.DocumentCommitProcessor.a(DocumentCommitProcessor.java:250)
at com.intellij.psi.impl.DocumentCommitProcessor.access$000(DocumentCommitProcessor.java:50)
at com.intellij.psi.impl.DocumentCommitProcessor$1.process(DocumentCommitProcessor.java:138)
at com.intellij.psi.impl.DocumentCommitProcessor$1.process(DocumentCommitProcessor.java:126)
at com.intellij.psi.impl.PsiDocumentManagerBase.finishCommitInWriteAction(PsiDocumentManagerBase.java:322)
at com.intellij.psi.impl.PsiDocumentManagerImpl.finishCommitInWriteAction(PsiDocumentManagerImpl.java:133)
at com.intellij.psi.impl.PsiDocumentManagerBase$3.run(PsiDocumentManagerBase.java:292)
at com.intellij.openapi.application.impl.ApplicationImpl.runWriteAction(ApplicationImpl.java:931)
at com.intellij.psi.impl.PsiDocumentManagerBase.finishCommit(PsiDocumentManagerBase.java:289)
at com.intellij.psi.impl.DocumentCommitThread$5.run(DocumentCommitThread.java:521)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:733)
at java.awt.EventQueue.access$200(EventQueue.java:103)
at java.awt.EventQueue$3.run(EventQueue.java:694)
at java.awt.EventQueue$3.run(EventQueue.java:692)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:703)
at com.intellij.ide.IdeEventQueue.e(IdeEventQueue.java:734)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:569)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:382)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
[ 11006] ERROR - .psi.impl.DocumentCommitThread - IntelliJ IDEA 14.1.4 Build #IU-141.1532.4
[ 11006] ERROR - .psi.impl.DocumentCommitThread - JDK: 1.7.0_60-ea
[ 11006] ERROR - .psi.impl.DocumentCommitThread - VM: Java HotSpot(TM) 64-Bit Server VM
[ 11006] ERROR - .psi.impl.DocumentCommitThread - Vendor: Oracle Corporation
[ 11006] ERROR - .psi.impl.DocumentCommitThread - OS: Windows 7
[ 11006] ERROR - .psi.impl.DocumentCommitThread - Last Action: EditorBackSpace
Your parser produced a tree which doesn't correspond to the new document text. It might be caused by incorrect incremental reparse. I'd start debugging in BlockSupportImpl and check how it calls your parser and what it makes of that text.
Is it possible to comment issue https://youtrack.jetbrains.com/issue/IDEABKL-6838 with link to a plugin page in repo? Cause I can't do it.
Here is the page: https://plugins.jetbrains.com/plugin/7796
Added to the description.
Inability to comment is strange. Could you please report that to YouTrack (JT) project tracker, attaching the screenshots showing that you're logged in and that you can't comment?
Reported. Thanks for an update.
Again, exception without my classes in the stacktrace. How to distinct if it's my fault or IDEA's? This time just click Tab/Ctrl-Tab in specific place:
Most likely it's some issue with incremental lexer-editor-highlighter update. The only way to tell is to reproduce and debug :(
I need to use my own skeleton file and JFlex Support allowes it, but Grammar-kit seems doesn't (or I just couldn't find how). Also, seems GK overrides JFlex Support's 'Generate lexer from Flex file' action. Created issue in GK repo, but no answer :(
Is there any workaround?
I have several inspections using same source data for analyzing, but these inspections shows different things and may be turned off separately..
This data generation is resourceful (well, not RESOURCEFUL, but weight something).
Anyway, would be nice to re-use this data in all related inspections or to make some kind of complex inspections which can be configured with checkboxes or smth to check this and not check this.
What is the proper way here?
There's no way currently unfortunately. Please vote for https://youtrack.jetbrains.com/issue/IDEA-142693
Got a problem:
Have working Perl bnf parser description.
Every statment must end up with semicolon.
Have a templating language HTML + embedded Perl. And in this templating language, rules slightly different: statement may end with semicolon or, template marker:
In Perl itself: print("Hello world");
In templating language with Perl: <% print(Hello world) %>
I could put few tricks in Perl parser itself and handle such markers there, but i belive it's not a good way.
Would be nice to somehow derive parser for templating language and just change rule for semicolon.
Is there any way for such situations?
Ok, still not sure what is the problem but found solution for myself. Not even found, more like guessed :) Happens sometimes :D
Problem:
We have a PsiElement, representing multi-line text block, which can be injected with other language.
PsiElement derived from PsiCommentImpl, because it should be ignored by parser while building a tree and basically it's a comment.
By default, if other language is injected into this PsiElement, hitting Enter inside it moves caret to the beginning of the block.
Problem solution:
overrided createLiteralTextEscaper of PsiElement
Created escaper class derived from CommentLiteralEscaper
Overrided isOneLine method of escaper and put return true; there.
But it's confusing, because according to documentation:
@return {@code true} if the host cannot accept multiline content, {@code false} otherwiseBut it can accept. Or I don't understand something.
Anyway, now it works as intended and if it's a bug, i'll be glad to add it to the JB tracker.
If it's not a bug, some explanation is welcome.
Working classes:
PsiElement: https://github.com/hurricup/Perl5-IDEA/blob/fa349ec05c547eba569483594312aede265013e9/src/com/perl5/lang/perl/psi/impl/PerlHeredocElementImpl.java
Escaper: https://github.com/hurricup/Perl5-IDEA/blob/fa349ec05c547eba569483594312aede265013e9/src/com/perl5/lang/perl/idea/intellilang/PerlHeredocLiteralEscaper.java
There's no standard way. But you surely can reuse the parser code by supplying it with some parameters.
Is it possible to stub leaf elements?
Until now i've stubbed only composite elements and it works fine. But same way doesn't work for leafs. Leaf elements being created using ASTFactory i've got only type and text there, but need ASTNode.
Where to dig? Or even better - example :)
No, it's not possible. You can stub their composite parents.
There's no such configuration option but you can always run JFlex manually with all required arguments
and / or setup a dedicated run configuration or an Ant task.
please create an issue in youtrack.jetbrains.com.
Thanks,
Done: https://youtrack.jetbrains.com/issue/IDEA-143464
Perl has so called variables interpolation. It's when you may use variables in regexps and strings:
At the moment in my plugin, string is a single token - STRING_CONTENT and interpolation doesn't work.
My idea of implementing interpolation is to lex string to several tokens and parse them as usual. Like: QUOTE-STRING_CONTENT-VARIABLE-QUOTE in this example.
But may be there is some specific mechanism in IDEA for such situations?
Yes, such strings should be treated as composite elements with multiple children. There's nothing special in it. You can see how it's done in Groovy support (e.g. GrString interface)
I have a composite PsiElement with stub. I'd like to stub some aggregated information from a subtree:
1. I'm looking for specific PsiElements in sub tree with PsiTreeUtil.findChildrenOfType()
2. I'm looking for specific leaf elements in PsiElements found in p.1, again with PsiTreeUtil.findChildrenOfType()
And here is the problem. When i'm editing file - everything is fine, works as intended.
But, when initial indexing is going on, PsiTreeUtil.findChildrenOfType() in p2 returns empty result to me. Don't know why.
I wrote pretty simple recursive searcher and it works fine. So tree is ok, but seems some tree properties or tree elements properties are not.
Basically, i've solved problem for now, but I'd like to know, where is the problem?
It's hard to tell from the information you give. It can be that you try to walk PSI tree from elementType.indexStub, where only stubs are available, not PSI. In that case, you can add all the necessary information to the stub when creating it from PSI.
And you can always use the debugger.
Sometimes so many things are going on on IDEA's side and requires a lot of time to understand what and how.
So I'm trying switch to other feature and ask here :)
Is there some kind of performance metric?
The problem is that my plugin begins to perform really badly on files larger than 100k.
I understand that my parser is FAR from ideal but i belive that architecture itself has some performance-based limitations like more than N kbytes, or K tokens, or X PsiElements.
I've checked IDEA's sources and there are only about 120 source files larger than 50kbytes and only 1k files larger than 20kbytes out of 46k files total.
Of course, it's because of Java concept itself, but i need to know where to stop looking for additional optimizations possibilities for my parser.
Would be nice to have some guildline for this.
UPD: checked 500k java file. Works like rocket. Seems something is bad with my parser. Or Perl itself. But anyway, would be nice to get an answer.
PSI tree structures are not well-suited for very large source code files, true. But the files have to be really large to not fit the memory. The best way to tell the reason of slowness is profiling, CPU sampling or memory snapshots. When you know exactly what's the problem, one can start thinking of possible solutions.