Perl5 plugin for Intellij IDEA

Hi everyone.

Recently i've decided to try to make perl5 plugin for InterlliJ IDEA. I've seen feew attempts to start, but they've failed after creating four base classes :)

The problem with perl is his too free syntax, which requires a very custom lexer and parser. I've tried to port perl's original lexer, but it's too big, got lot of legacy stuff, so i've droped the idea.

Currently, i'm making lexer with JFlex and it works, but need tunings for different perlish situations. Anyway, some syntax is already highighted:
46fdd3eb65.jpg
The second problem is that language plugins development documentation looks outdated (of course, not so many readers), but examples helps in such cases. Sometimes...

But is there some gurus in language plugins who could help with tricky questions or just to save some time in useless tries?

Also, if you would like to participate in perl5 plugin development, you are welcome: https://github.com/hurricup/Perl5-IDEA

1
236 comments
> SegmentArray.segmentNotFound
I ran into a similar exception (although different assertion error: 6, not 2005). In this case it was caused by lexer returning null element type in the middle of the file (my fault).
0
Avatar
Alexandr Evstigneev

Same with me, i mean - my fault.
Just implemented debugging lexer wrapper and dumped tokens with proper example and incorrect one. Mistake became obvoius.

0
Avatar
Alexandr Evstigneev

Got the performance issue with references search for functions with common names, like constructor: new. It's VERY common for OO perl project.
As far as I understand the process, IDEA takes all 'new' occurances from find usages index and than checks every single one - if it's references to the function i'm searching for.
The second problem here is that Perl is not strongly typed, so reference resolution may be a bit heavy and tons of them - just HEAVY.
I can see here the only way to stub functions invocations with their name and type, and then redefine references search algoythm.
But index would be a really large (only definitions index is 11mb already on not very large project). There are more side-effects, but this is my main concern.
Is it a good way to solve the problem or there is something else? Because it's even acceptable when you are finding usages for a single function but total disaster on unused functions inspection.

0

Other languages are suffering from this issue as well. We don't have a good solution yet. If you're able to build an index that speeds up the search (and it doesn't take hundreds of megabytes), that's great and by all means do it. For "unused" highlighting in the editor, we have an approximate algorithm that doesn't search for usages when there are too many candidates (see com.intellij.psi.search.PsiSearchHelper#isCheapEnoughToSearch).

0
Avatar
Permanently deleted user

Hi,

I have done this exactly in this way and IDEA itself seems to do it for instance for java-doc comments where they separate comment-data, comment-leading-asterix, doc-tags, etc.. when you have a comment like this

/**
* Provides a processor to collect all variables in the current scope to suggest them for autocompletion.
*
* @author patrick (5/22/13)
*/


I used it in the plugin for Mathematica to extract so called "named characters" which are e.g. greek symbols in Mathematica notation. You can see them here

http://i.stack.imgur.com/spMpc.png

I wanted to be able to handle them differently (even in strings) because I have implemented a code folding that lets you fold them to make real greek letters appear

http://i.stack.imgur.com/mG1LY.png

So what you need to do is

  1. adapt your lexer (look for instance here in line 218 where I included the named characters in the string scanning)
  2. create the additional IElements that you need
  3. change your parser to parse the additional token correctly (see here in line 56)


That's about it and then you can use it. In general (I'm pretty sure I don't have the true view on the whole subject) I would suggest to use the lexer to separate tokens if possible. You could extract the same information on a PsiElement that contains the whole string but this would be more tedious.

Cheers
Patrick

0
Avatar
Alexandr Evstigneev

Implemented strings content as a lazy parsable elements. Works like a charm.

0
Avatar
Alexandr Evstigneev

Inevitably i'm coming to release of version 1.0 of Perl5 plugin. And I really want to release it this week.
But there is a little problem. My design mistake from the very beginnig. The problem is that i've used some IDEA specific classes and now plugin doesn't work as intended or even doesn't work at all in the specific IDEs: PyCharm, RubyMine and so on.
Downloads static says that would be nice to plugin work in them.
I've localized problematic classes: https://github.com/hurricup/Perl5-IDEA/issues/265
And of course, I could dig and solve this problem myself, but it will take precious time and I won't be able to release this week and this is really important.
So, i need help. The explanation how to implement these features cross-IDE. The more detailed explanation will be - the better.

Thanks in advance.
0
Avatar
Alexandr Evstigneev

I've just released first version of Perl5 plugin. Thanks to everyone for help!

0
Avatar
Alexandr Evstigneev

Formatting and indenting being done in  TemplateState#smartIndent and TemplateState#reformat
And I don't see how I can do something there.

0
Avatar
Alexandr Evstigneev

There is a nice EP IndentStrategy which allowes me to suppress indeting of here-doc elements. But there is a little inconvenience. For example:

my $somevar = <<'MARKER';
... heredoc content...
MARKER

Here-doc content is a lazyparsable element.
I'd like to suppress indenting of content and closing marker if selection includes whole code block, but want to allow indenting of content if selection is inside it. And seems there is no way to do it, because canIndent takes only one argument - element. I should suppress/allow content indenting in any case.
Just a thought, may be I should make a feature request or something.

0

I guess only end MARKER should keep its position. Thus it's the only PSI element which should never be indented. When indenting lines, the editor checks every element at the beginning of every line which potentially should be indented. Since other lines will not contain the end marker, it should be sufficient to place a code like this:

 
public boolean canIndent(@NotNull PsiElement element) {
  return element.getNode().getElementType() != PerlTokenTypes.END_MARKER;
}


I used PerlTokenTypes.END_MARKER which you can replace with the actual token type I'm not aware of. Note that internally PsiElement.findElementAt(offset) method is used to find the element at line start. It goes deep enough till a leaf element is reached. In your case it should be END_MARKER or whatever the actual type is.

0
Avatar
Alexandr Evstigneev

Yes, i've already done it just for an end marker. The problem is that I can't implement more complicated behavior. Thanks anyway :)

0

Hi,

The most proper way to handle this is to implement formatter that will be able to handle this case and don't break code during formatting.

As a workaround you can disable indenting/formatting for particular template:

– in runtime via Template#setToReformat/Template#setToIndent (you may want to use TemplateOptionalProcessor for modifying corresponding templates)
– in live templates xml via toReformat/toIndent attributes of 'template' tag

0
Avatar
Alexandr Evstigneev

Seems that there is no toIndent attribute in livetemplate tag, only toReformat, but yes, optionalProrcessor solved problem. Thanks.

0
Avatar
Alexandr Evstigneev

I've tried to implement all string constants as lazy parsable elements, but this change has killed performance (because there are lot of strings in perl scripts).
Is there a way to speed things up, like re-use lexers and parsers and not create them for each LP element? They are not thread-safe so not possible to just make a static lexer/parser.

0

Have you profiled it? What exactly is slow?

0
Avatar
Alexandr Evstigneev

Nope, not yet. I belive problem is that there are hundreds of them in a file.

0
Avatar
Alexandr Evstigneev

Was my bad. Sorry.

0
Avatar
Alexandr Evstigneev

Got a problem. Implemented my own VariableInplaceRename handler and added it into EP renameHandler
But, when i'm trying to rename an element, getting
http://dl1.joxi.net/drive/0004/3351/294167/150920/99c4cc8f96.png
And I'm not sure how to get rid of it.
Pre-history: the reason for this to implement my own VariableInplaceRenamer and suppress VariableInplaceRenamer#collectAdditionalElementsToRename
Pre-pre history: a lot of name identifiers are strings. And when i'm trying to refactor such elements, getting this (because of method above):
http://dl1.joxi.net/drive/0004/3351/294167/150920/d1d910c60f.png
And i don't really want to change string to something else in parser.

0

From the pre-pre-history:
'Non-code occurrences' message appears when there is no reference at place with the same text but without reference. Is it correct that there should not be reference?

Anna

0
Avatar
Alexandr Evstigneev

Yes, this is correct. Illustration:

http://dl2.joxi.net/drive/0005/1935/382863/150921/11381406ac.jpg
So we have two similar blocks with different queries. Attempt to refactor second usage offers to refactor it's declaration and string inside the first declaration not actually related to the second usage.

0

Did you consider to disable standart inplace rename with com.intellij.lang.refactoring.RefactoringSupportProvider#isInplaceRenameAvailable?

0
Avatar
Alexandr Evstigneev

Nope. I want to have it. The only way I figured out: to change string in parser to something else. But don't like this way.

0

So just disable inplace refactorings for your language and provide your own inplace refactoring handler instead. Please check what happens if user disables inplace refactorings in editor settings (File|Settings|Editor|General|Refactorings)

0
Avatar
Alexandr Evstigneev

Ok, i'll check this way. Thank you.

0
Avatar
Alexandr Evstigneev

Just a second. Isn't that what i've done on the first screenshot? VariableInplaceRenameHandler ?

0

If you suppress inplace renamer for your language in RefactoringSupportProvider, then corresponding handler won't be suggested. What screenshot do you mean? I am not sure that I understand the problem now.

0
Avatar
Alexandr Evstigneev

I ment this one
http://dl1.joxi.net/drive/0004/3351/294167/150920/99c4cc8f96.png
Ok, i've disabled variable inplace renaming with RefactoringSupportProvider. And now it works with MemberInplaceRenameHandler and works as I wanted, thanks.
But, now I don't understand what is going on in there (well i can dig but would be nice to have small explanation) :)

1. Why there are Variable and Member handlers? What the difference?
2. Is there way to avoid situation on the screenshot?

0

MemberInplaceRenameHandler? Instead of yours? That's strange.

The goal of the dialog is the ability for the user to resolve ambiguity: mainly it was introduced to distinguish between module/directory rename from Project View when module content root is selected. It should appear only when IDE can't deside byitself which handler is correct.

Looks like the way to implement your own inplace variable rename handler is not very convenient, I'll try to make this mess more clear but I am afraid not in IDEA 15.

0
Avatar
Alexandr Evstigneev

Sorry, my bad, forgot to override VariableInplaceRenameHandler#isAvailable so my handler was disabled together with default one :)
Thanks again.

0

Please sign in to leave a comment.