Code Completion With Special Characters

Do you mean autopopup completion that doesn't appear, or explicit invocation (Ctrl+Space) that doesn't work?

Neither of these work

Is getVariantsFromContributors called during explicit completion?

Thanks, when I type a special character, getVariants is called with explicit completion, but not with auto completion. If I type any character other than a number or letter, auto completion doesn't happen and getVariants isn't called. Furthermore, using a PsiViewer, I see that the '$' is the element it should be (expression_start), however in getVariants the element type is BAD_CHARACTER. Once the expression start has been typed and I do explicit completion, the element type in get variants it the next part of the expression (key), giving me completion for possible keys (as it should).

By default, autopopup completion is only started when letters or digits are typed. You can add your own TypedHandlerDelegate and handle $ there.

The part about element types is hard to understand without knowing the details of your language and parser, but it might have something to do with the dummy identifier being inserted by completion engine.

So the auto popup is now working with the typed handler, I still have problems with the elements. In debugger I see PsiErrorElement:MyTokenType.KEY expected, got 'IntellijIdeaRulezzz'. Before i got just the letter I, is this the dummy identifier you mean? 

Yes, IntellijIdeaRulezzz is the dummy identifier, and your parser should be prepared to it, as any other text.

Ah ok, so how can I set my parser to be prepared to it? The parser understands regular text, in the first element there is no text however, therefore it returns an error if text is returned. Is this the issue?

The parser surely can mark this as an error, but it still should produce some PSI for it that your completion would be able to analyze, and generate suggestions based on that PSI structure.

Sorry I'm not sure I follow. I believe the parser returns the expected element, as when I start typing the start of the expression, the psi viewer recognises this as the expression start. Is this what you mean when you say the parser should produce some element that completion can analyze?

What do you mean by "expected"? Expected by whom for which purpose?

The parser should produce some tree for every text, including the one with a dummy identifier IntellijIdeaRulezzz. In completion, you inspect the resulting tree (starting from completionParameters.getPosition(), which will likely contain the dummy identifier) and generate relevant suggestions for this position. The dummy identifier is inserted only for making it easier to inspect the parse tree.

Sorry, I mean expected by the parser. I've been trying to solve this however my progress is now stopped by an out of memory issue. The OOM error occurs whenever code completion is invoked. Originally the Xmx was set to 500m and as soon as I typed anything it would run out of memory. Testing it out, I set it to 1G, 1.5G and even 3G and it had enough memory so I could type (with no code completion showing up), but as soon as I explicitly invoke code completion, completion popup shows a loading symbol until it eventually runs out of memory. I'm really unsure how this has happened, any ideas or pointers on how I can debug would be amazing.

Edit: In debugger, I see that getVariantsFromContributors isn't being called anymore. It runs out of memory before this.

There's probably a memory snapshot generated in IDE bin or user home directory, you can try to understand the reasons from it. In addition, OOM could be caused by stack overflow, so a stacktrace of the OOM might also provide clues.

Thanks for replying. I've looked in home/user/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-0/191.7479.19 (which I am using). In the bin folder, I don't see any kind of memory snapshot. I'm not totally sure what I'm looking for or if I'm in the right place. Also, when I get the OOM, there is no stacktrace provided within the IDE. It shows the window with the options shutdown, continue or memory dump. I also don't know what to do with the memory dump or if this is useful to me. Where can I find the stacktrace or memory snapshot? Sorry I'm still getting to grips with stuff like this, thank you for your patience.

I've gotten to the bottom of the OOM error. I had removed the line 

[^] { return BAD_CHARACTER; }

somewhere along the way, which was causing this out of memory issue. So returning to the original problem, I think my issue is with the JFlex macros. When I have:

KEY_CHARACTER= [^] | [${]

, with the expression start being ${, code completion works. However, this also works for any other character, which I dont want, I only want to to complete for ${ and otherwise return bad character.  So I believe the way to achieve this is to write:

KEY_CHARACTER= [${]

however with this, the psi elements work correctly, but code completion doesn't work. Can someone see what I am doing wrong here? getVariantsFromContributors is being called in both cases, but my lookup element isn't being called with the 2nd case. Any help would be great, thank you.

Sorry, this is too unspecific, I can't understand what you mean by "this also works for any other character" (what's "this", what's "works for", where the characters are), "I only want to to complete for ${ and otherwise return bad character" (what's "complete for", "return" from where), "psi elements work correctly" (PSI elements usually don't work, they're just the data different code works with), "my lookup element isn't being called" (what exactly isn't called that you expect to be?).

Very sorry, I will try be more specific. So I want to be able to have code completion for an expression like this - ${value}, where "${" is the expression start, "value" is the value, and "}" is the expression end. My end goal is for the user to be able to type "$", and code completion will present "${}". So to achieve this, I have tried using the JFlex macro:

EXPRESSION_START= [^] | [${]

With this macro, when I type "$", code completion is invoked and shows ${}, which is good. The issue with the macro, is that when I type something like "hello", instead of "${", the syntax highlighter shows it to be a correct expression start, which it is not. I want only "${" to be shown as an expression start, any other characters, such as "hello", should be returned as bad characters. So in attempt to fix this, I replace that macro with: 

EXPRESSION_START= [${]

With this macro, when I type "$", code completion is not invoked. However the other issue is fixed, as when I type "hello", it is shown as bad characters (as it should be), and "${" is shown as expression start. I am trying to find the right macro, which will allow code completion to be invoked, and only allow "${" as the expression start. 

The second rule seems correct. How would "$hello" be lexed/parsed with it? Do you use MultiplePsiFilesPerDocumentFileViewProvider?

"$hello" is incorrect with the second rule. "$" is shown as correct, but the "h" in hello is parsed as bad character. I'm not totally sure about MultiplePsiFilesPerDocumentFileViewProvider. To perform code completion, I extend CompletionContributor, and use the "extend" method in my completion class' constructor. Is this the information you are after? To be more concise and specific with the issue, if the rule starts with "[^] |", code completion is invoked explicitly and automatically. Otherwise, it isn't invoked explicitly or automatically. This only applies to the EXPRESSION_START rule, which I believe is because it is the first element to be typed on any line. Thank you for your help and patience.

If you don't know what MultiplePsiFilesPerDocumentFileViewProvider is, you' probably aren't using it :)

So, if "h" is parsed as a bad character, how is "e" parsed then?

"e" and "llo", are parsed as the value. So it is "$h" shown as expression start, and "e" is a bad character and "llo" is correct value characters. The kind of functionality I'm after is similar to that of a java keyword, where you can be at the start of an empty file, invoke code completion and see a list of keywords. I can't see what I am doing wrong, if I make a sample project, could you see if you can spot my mistake?

If you want your completion to work after $, you need to write a pattern for the bad character that comes after a $, because CompletionParameters#getPosition will return this bad character.

I want code completion to work before the $, as if it was a java keyword. In java files, you can invoke completion right at the start of the file and it lists keywords. If you start typing a keyword, it also lists completion options. This is what I want, however I can only achieve it if I use the macro:

EXPRESSION_START=[^] | [${]

This is causing the bad character issue, because the "^" represents any character. But, it allows code completion to be invoked at the start of the file. So, when I use the correct macro:

EXPRESSION_START=[${]

All is fine with the bad character, only "${" is accepted, but code completion isn't invoked at the start of the file, as it was with the "^". My main question is, what is the difference between using "^" when it comes to code completion? Why does it either allow or not allow completion to be invoked?

Furthermore, the completion after the expression start is working. For example for the expression "${hello}", i can type "${h", which is shown as correct characters, and "h" invokes code completion for "hello". I can also make the expression start only accept "${", with anything else being bad characters. The issue is it either accepts only "${", or has code completion. I struggle to get both working. 

If you want code completion to work in the start of a file, you should check how a file starting with "IntelliJ" is parsed, and write a pattern for that PSI.

"^" is irrelevant for code completion. Most likely, your parser shouldn't be changed for the code completion to work. It just has to produce some tree for every input text, and then it's CompletionContributor's responsibility to analyze all patterns you care about and check whether the current PSI position (with dummy identifier inserted) matches any of those.

Thank you very much, the issue is now mostly resolved after putting code completion for BAD_CHARACTER. This now gives me a new problem though. I type "$", code completion shows "${}" (which is great). When I hit enter, it inserts the whole "${}", leaving me with "$${}". When I look at the elements present. The first element is actually "PsiErrorElement:EXPRESSION_START expected, got "$"", with a sub element of "PsiElement(BAD_CHARACTER)". The following characters are just BAD_CHARACTERS, without an error element. My initial thought is to just write code completion for the error element but there are two issues. I don't create the error element in my code, and I can't get an instance of it. Second, if I wrote completion for the error element, wouldn't it always popup every time a mistake is made? Any guidance here would be much appreciated.

$${} is probably inserted because your LookupElement's lookup string is "${}", but the prefix matcher doesn't include "$" as prefix. You can fix it by either using CompletionResultSet#withPrefix, or by inserting only "{}".

And your completion contributor shouldn't react to any error element or BAD_CHARACTER. You can write a custom condition that checks the surrounding of that error element and determines whether completion makes sense there, and what kind of it does.

Excellent, everything is working now with this prefix handler. Thanks a lot for your help.