I am new to intellij plugin writing. I started writing a intellij plugin for one of our custom language. I am following the tutorial given in intellij official site. Also I downloaded their Grammar-Kit repo from github to understand the code base.
While writing my plugin I am using jflex for Lexer and bnf grammar for parser. I find difficulty to implement a lexer to send proper tokens to the parser.
What I see in the Grammar-Kit repo that Lexer (for bnf) is pretty simple and it is only recognizing String, Number, Id, white space, comments and special characters('(', '*' etc.). It does not recognize bnf keywords like 'private', 'external', 'meta' etc.
Now when I am seeing the PSI tree of a example bnf file say for the line,
private myRule ::= '(' myExpression ')' ';'
is as below,
BNF_SEQUENCE: '(' myExpression ')' ';'(19,43)
What i am seeing here is 'private' is recognized as
PsiElement.id by the Lexer, after that some code wrap it with a BnfModifier object which is declared the bnf file The same for BNF_SEQUENCE, BNF_REFERENCE_OR_TOKEN, BNF_STRING_LITERAL_EXPRESSION etc. They are all not recognized by Lexer, however some code wrap them with bnf file declared objects.
I want to understand how this wrapping is done on top of Lexer recognized tokens. It will help me to recognize keyword in our DSL and highlight them differently, auto-complete them etc.