Custom language highlighting in new file Follow

Please share full sources

Is there a paid support available somewhere as I can't share the full sources publicly...

Please use https://uploads.services.jetbrains.com/ and write filename here

Contains the full sources + bnf and lexer files seperately: Upload id: 2021_01_14_4uHJTrYk3jfiSCBw (files: CompositeSteps.bnf and 2 more)

Unfortunately I cannot build and test the plugin due to missing dependencies, so I can only "guess" a bit and hopefully give some useful recommendations on where to look for bugs.

Given that "Composite" is keyword and should be highlighted via CompositeStepsTypes.COMPOSITE_KEYWORD in syntax highlighter, the initial place to search is lexer.

- Lexer has multiple states, are you sure switching them and especially restoring YYINITIAL state works 100%?

- if possible, reduce total number of states in Lexer

- Lexer should be tested more extensively, see also com.intellij.testFramework.LexerTestCase#checkCorrectRestart(java.lang.String)

Thanks for your suggestions, unfortunately, they didn't really tell me why it's not working. I'm certainly happy about the suggestions of using LexerTestCase as we struggled with writing test cases for the lexer before (I don't think its mentioned in the how to write a custom language tutorial).

In the end, I have now boiled it down to a single line that makes everything work/break:

import com.intellij.lexer.FlexLexer;
import com.intellij.psi.tree.IElementType;

import static com.intellij.psi.TokenType.BAD_CHARACTER;
import static com.intellij.psi.TokenType.WHITE_SPACE;

%%

%class CompositeStepsLexer
%implements FlexLexer
%unicode
%function advance
%type IElementType
%eof{  return;
%eof}

WHITE_SPACES=\s+

COMPOSITE_KEYWORD="Composite:"
WORD=[\S]+

%%
<YYINITIAL> {
  {WHITE_SPACES}                  {  return WHITE_SPACE; }
  {COMPOSITE_KEYWORD}             {  return COMPOSITE_KEYWORD; }
  //This next line is important for it to work
  {WORD}                         { yybegin(YYINITIAL); return WORD;}
}

[^] { return BAD_CHARACTER; }

This version correctly immediately highlights the word "Composite:" once I type it into a blank file, commenting out the last instruction in the <YYINITIAL> state breaks the behaviour. Without this instruction, I have to re-open the file for the highlighting to work.

My plugin will work with this additional rule as the bnf file will warn the user that an invalid token WORD has been generated, but 1) I would like to understand why I need this line and b) I would like a test case that ensures this works as right now I have not been able to write a test case that allows me to simulate typing something into the IDE and then verifies the highlighting (not annotations).

Many thanks for your help!