I have implemented a lexer to use in my `parserDefinition` and `syntaxHighlighterFactory` extension points. It isn't a state-based lexer. It is considerably more sophisticated than your usual lexer, and it is rather slow, so it is vital to use incremental reparsing.
The lexer does recognize that there are certain points where it can be safely restarted in a "default"/zero state.
So what the lexer does is claim that it is in state 0 at those points, and claim to be in some other, unique state at any other point. Unfortunately, the way the lexer is being used by the PsiBuilderImpl (for parsing) and the LexerEditorHighlighter (for Highlighting) are not quite how I expected.
PsiBuilderImpl doesn't seem to do incremental relexing at all! It just asks for the whole document to be relexed every time. Am I missing something? Is it possible to get incremental relexing to happen for parsing without reimplementing (parts) of PsiBuilderImpl?
LexerEditorHighlighter does do somewhat incremental relexing, but it seems to have a problem knowing when to stop. I expected that if I insert a character, the highlighter will relex from the previous offset marked with the zero state. Indeed this seems to be happening.
But I would also expect that, if at some point it receives a token that is in the zero state, and that token was *already* in the zero state prior to me inserting the character, then the highlighter should *not* request any further tokens (obviously they will be all the same as before, since the tokenizer is in the same state in the same position of the file which is unchanged after that point). This is not what is happening.
What seems to be happening is that the tokenizer asks for a relexing of the text from the previous zero-state position up, up to the next zero-state position appearing at the beginning of a line. I assume that I need to change the LexerEditorHighlighter, is this correct?
Thank you for your attention,