Custom language support w/o JFlex

Hi. As I work towards integrating ANTLR into intellij, I'm slowly morphing the custom language support example.  This first iteration removes the dependency on JFlex and instead uses my own simple brain-dead handbuilt lexer. It is mostly compatible with the specification in the .flex file but was mainly I used it to figure out Intellij's API. I've attached the simple complete zip.

BTW, there is an ambiguity in the API documentation.  For method Lexer.start(), it says:

   * @param endOffset    offset to stop lexing at

But I have found that I expected it to be the last good offset when in fact it is the first bad offset ;).   In other words it is the length of the buffer not the last offset. it is not incorrect, just ambiguous. It should say something like "The last valid index + 1" or something, which is awkward but unambiguous. I would also like to know when this is ever called with something other than a start offset of zero and end offset of len(buffer).

Next step, remove the dependency on the .bnf parser file.

Comment actions Permalink

Hi Terence,

IntelliJ uses a couple of different types of lexer - you can use different lexers for the parsing process and the syntax highlighting process, for example. Lexers can be called to incrementally re-lex a part of a file after editing to highlight it, which is when it would be called for part of the buffer. However parsing only happens for the entire buffer so the lexer used in that context will only be called like that.


Comment actions Permalink

Thanks Colin!  Got my parser working. will post soon.


Please sign in to leave a comment.