[Custom Language Plugin] Highlighting not working despite correctly parsed psi file



i have implemented a custom language plugin with a parser and a lexer.

Following problem:
When i edit the file, the highlighting doesnt work for the new tokens. The code is parsed correctly (green checkmark). If i reload the file or view the psi tree, the highlighting gets updated.

I tried to force rehighlighting(temporary via an action), but it doesnt work.



Most probably a problem with (incremental) lexing. Impossible to tell without looking at actual sources.


Yann Cebron


Would be nice, if you take a quick look. Some tokens seem to be just fine like PNAME_NS from my .flex . I think only the highlighting of keywords is broken. I am currently searching on github for flex files of other language plugins.




Reproduced. There is definitely a problem with recognizing keywords on lexer level:

You can use internal action "Show Editor Highlighter Tokens" to get this special highlighting in debug instance.

Unfortunately I can't spot the problem in short time. I suggest adding a number of com.intellij.testFramework.LexerTestCase to make sure your lexing works 100% as expected and does not yield BAD_CHARACTER tokens for valid (but incomplete) input.

A list of all OSS plugins with custom language as reference https://plugins.jetbrains.com/intellij-platform-explorer?extensions=com.intellij.lang.parserDefinition



Yann Cebron 

Ok i will do that. If it has something to do with the lexer, i find it a bit weird that the parser seems to get the right tokens from the lexer (no parsing errors), but the highlighting seems to be stuck with the bad_character tokens.


Got a solution (i guess?)

This seems to be the same issue as in:


If i add:

{WORD}  { yybegin(YYINITIAL); return BAD_CHARACTER;}

to the flex. The Keywords are correctly highlighted after completing them. I roughly went through some flex files on github, but those just use:

[^] { yybegin(YYINITIAL); return BAD_CHARACTER; }

According to the jflex tutorial. This method is only for improving performance for backtracking especially of keywords. https://jflex.de/manual.html#performance

"keyword1"  { return symbol(KEYWORD1); }
"keywordn"  { return symbol(KEYWORDn); }
[a-z]+      { error("not a keyword"); }

Also i am still a bit confused why the parser gets the keyword token, but the highlighter doesnt get the keyword token and even marks the last char as a bad character. 


Glad you got it working.

From https://plugins.jetbrains.com/docs/intellij/implementing-lexer.html:

Lexers, and in particular JFlex-based lexers, need to be created so that they always match the entire contents of the file, without any gaps between tokens, and generate special tokens for characters which are not valid at their location. Lexers must never abort prematurely because of an invalid character.