How to wrap composite PsiElement on top of Lexer recognized elements in Grammar-Kit ? Follow

Hi All,

I am new to intellij plugin writing. I started writing a intellij plugin for one of our custom language. I am following the tutorial given in intellij official site. Also I downloaded their Grammar-Kit repo from github to understand the code base.

While writing my plugin I am using jflex for Lexer and bnf grammar for parser. I find difficulty to implement a lexer to send proper tokens to the parser.

What I see in the Grammar-Kit repo that Lexer (for bnf) is pretty simple and it is only recognizing String, Number, Id, white space, comments and special characters('(', '*' etc.). It does not recognize bnf keywords like 'private', 'external', 'meta' etc.

Now when I am seeing the PSI tree of a example bnf file say for the line,

private myRule ::= '(' myExpression ')' ';'

is as below,

BnfFile:Dummy.bnf(0,43)
  BNF_RULE:myRule(0,43)
    BNF_MODIFIER(0,7)
      PsiElement(id)('private')(0,7)
    PsiWhiteSpace(' ')(7,8)
    PsiElement(id)('myRule')(8,14)
    PsiWhiteSpace(' ')(14,15)
    PsiElement(::=)('::=')(15,18)
    PsiWhiteSpace(' ')(18,19)
    BNF_SEQUENCE: '(' myExpression ')' ';'(19,43)
      BNF_STRING_LITERAL_EXPRESSION: '('(19,22)
        PsiElement(string)(''('')(19,22)
      PsiWhiteSpace(' ')(22,23)
      BNF_REFERENCE_OR_TOKEN: myExpression(23,35)
        PsiElement(id)('myExpression')(23,35)
      PsiWhiteSpace(' ')(35,36)
      BNF_STRING_LITERAL_EXPRESSION: ')'(36,39)
        PsiElement(string)('')'')(36,39)
      PsiWhiteSpace(' ')(39,40)
      BNF_STRING_LITERAL_EXPRESSION: ';'(40,43)
        PsiElement(string)('';'')(40,43)

What i am seeing here is 'private' is recognized as PsiElement.id by the Lexer, after that some code wrap it with a BnfModifier object which is declared the bnf file The same for BNF_SEQUENCE, BNF_REFERENCE_OR_TOKEN, BNF_STRING_LITERAL_EXPRESSION etc. They are all not recognized by Lexer, however some code wrap them with bnf file declared objects.

I want to understand how this wrapping is done on top of Lexer recognized tokens. It will help me to recognize keyword in our DSL and highlight them differently, auto-complete them etc.

Thanks,

Subhojit