I was wondering if there was any documentation regarding how to create context-sensitive tokens. I'm trying to write a grammar for a language that supports preprocessor directives, one of which has a regex like: #pragma deprecated( .+)?(\r|\n|\r\n)
In plain text: "#pragma deprecated" is required, and either EOL, EOF, or a token containing the remaining characters on the line is expected after.
#pragma deprecated This is an example
'#' 'pragma' 'deprecated' 'This is an example'
The problem I'm running into is that while tokens are generated fine for '#'. "pragma" and "deprecated", the optional string at the end is causing me trouble, since if I have a token with the regex ".+(\r|\n|\r\n)" and make that token optional, then that token is generated for the entire lexer (which makes sense). I also tried prepending a positive lookbehind assertion to the 'math until eol' regex, which does't seem to be supported (regex:(?<="deprecated")).
Is there a way I can embed some logic to generate this token contextually within the bnf grammar itself? I believe this is possible within JFlex, but I wanted to know if this were possible within the bnf grammar file.
If I can't, then I'm thinking my best alternative is to make a regex for all preprocessor directives "^#.*$" and then parse that regex manually by passing the token to determine validity. I think I could then generate syntax highlighting by making the preprocessor an embedded language.
I read in the change log notes that there is support for empty tokens (i.e., tokens without a string:
tokens = [
Does anyone know of an example of how to use this feature? I think this might be another answer to my problem.