[Custom Language Plugin] Highlighting not working despite correctly parsed psi file

Most probably a problem with (incremental) lexing. Impossible to tell without looking at actual sources.

Yann Cebron

https://github.com/marcelhillesheim/outSPARQL
https://github.com/marcelhillesheim/outSPARQL/blob/master/src/main/java/language/_SparqlLexer.flex

Would be nice, if you take a quick look. Some tokens seem to be just fine like PNAME_NS from my .flex . I think only the highlighting of keywords is broken. I am currently searching on github for flex files of other language plugins.

Reproduced. There is definitely a problem with recognizing keywords on lexer level:

You can use internal action "Show Editor Highlighter Tokens" to get this special highlighting in debug instance.

Unfortunately I can't spot the problem in short time. I suggest adding a number of com.intellij.testFramework.LexerTestCase to make sure your lexing works 100% as expected and does not yield BAD_CHARACTER tokens for valid (but incomplete) input.

A list of all OSS plugins with custom language as reference https://plugins.jetbrains.com/intellij-platform-explorer?extensions=com.intellij.lang.parserDefinition

Yann Cebron 

Ok i will do that. If it has something to do with the lexer, i find it a bit weird that the parser seems to get the right tokens from the lexer (no parsing errors), but the highlighting seems to be stuck with the bad_character tokens.

Got a solution (i guess?)

This seems to be the same issue as in:

https://intellij-support.jetbrains.com/hc/en-us/community/posts/360010287040-Custom-language-highlighting-in-new-file?page=1#community_comment_360002658559

If i add:

WORD=[a-zA-Z0-9]+

{WORD}  { yybegin(YYINITIAL); return BAD_CHARACTER;}

to the flex. The Keywords are correctly highlighted after completing them. I roughly went through some flex files on github, but those just use:

[^] { yybegin(YYINITIAL); return BAD_CHARACTER; }

According to the jflex tutorial. This method is only for improving performance for backtracking especially of keywords. https://jflex.de/manual.html#performance

"keyword1"  { return symbol(KEYWORD1); }
..
"keywordn"  { return symbol(KEYWORDn); }
[a-z]+      { error("not a keyword"); }

Also i am still a bit confused why the parser gets the keyword token, but the highlighter doesnt get the keyword token and even marks the last char as a bad character. 

Glad you got it working.

From https://plugins.jetbrains.com/docs/intellij/implementing-lexer.html:

Lexers, and in particular JFlex-based lexers, need to be created so that they always match the entire contents of the file, without any gaps between tokens, and generate special tokens for characters which are not valid at their location. Lexers must never abort prematurely because of an invalid character.