Custom language highlighting in new file



I have an issue with my custom language highlighting. Everything works in an existing file, but if I create a new file and start typing, the parsing seems to not start and there is no highlighting. However, if I open the PSI structure, the PSI structure viewer immediately displays the correct highlighting. Also, if I close and re-open the file, the highlighting also appears.

The following GIF shows the issue:

Any tips what might cause such an issue?

Comment actions Permalink

Please share full sources


Comment actions Permalink

Is there a paid support available somewhere as I can't share the full sources publicly...

Comment actions Permalink

Contains the full sources + bnf and lexer files seperately: Upload id: 2021_01_14_4uHJTrYk3jfiSCBw (files: CompositeSteps.bnf and 2 more)

Comment actions Permalink

Unfortunately I cannot build and test the plugin due to missing dependencies, so I can only "guess" a bit and hopefully give some useful recommendations on where to look for bugs.

Given that "Composite" is keyword and should be highlighted via CompositeStepsTypes.COMPOSITE_KEYWORD in syntax highlighter, the initial place to search is lexer.

- Lexer has multiple states, are you sure switching them and especially restoring YYINITIAL state works 100%?

- if possible, reduce total number of states in Lexer

- Lexer should be tested more extensively, see also com.intellij.testFramework.LexerTestCase#checkCorrectRestart(java.lang.String)

Comment actions Permalink

Thanks for your suggestions, unfortunately, they didn't really tell me why it's not working. I'm certainly happy about the suggestions of using LexerTestCase as we struggled with writing test cases for the lexer before (I don't think its mentioned in the how to write a custom language tutorial).

In the end, I have now boiled it down to a single line that makes everything work/break:

import com.intellij.lexer.FlexLexer;
import com.intellij.psi.tree.IElementType;

import static com.intellij.psi.TokenType.BAD_CHARACTER;
import static com.intellij.psi.TokenType.WHITE_SPACE;


%class CompositeStepsLexer
%implements FlexLexer
%function advance
%type IElementType
%eof{ return;



//This next line is important for it to work

{WORD} { yybegin(YYINITIAL); return WORD;}

[^] { return BAD_CHARACTER; }

This version correctly immediately highlights the word "Composite:" once I type it into a blank file, commenting out the last instruction in the <YYINITIAL> state breaks the behaviour. Without this instruction, I have to re-open the file for the highlighting to work.

My plugin will work with this additional rule as the bnf file will warn the user that an invalid token WORD has been generated, but 1) I would like to understand why I need this line and b) I would like a test case that ensures this works as right now I have not been able to write a test case that allows me to simulate typing something into the IDE and then verifies the highlighting (not annotations).

Many thanks for your help!




Please sign in to leave a comment.