I have a lexer which uses a stack to manage its states. I think this is a common pattern and also suggested by https://intellij-support.jetbrains.com/hc/en-us/community/posts/360000594004/comments/360000196484. Unfortunately, this seems to break due to the following limitation.
An essential requirement for a syntax highlighting lexer is that its state must be represented by a single integer number returned from Lexer.getState(). That state will be passed to the Lexer.start() method, along with the start offset of the fragment to process, when lexing is resumed from the middle of a file. -- https://plugins.jetbrains.com/docs/intellij/implementing-lexer.html
I cannot represent a stack of states with one integer. I looked up implementations of other language plugins, but I couldn't find a good solution.
The RegExpLexer of IntelliJ and the elm-plugin (https://github.com/durkiewicz/elm-plugin) seem to ignore the requirement and don't do anything to restore the full state when `Lexer.start` is called.
The Dart plugin deletes any remaining data of previous executions from the stack. I guess this at least ensures deterministic behavior, but the state is still not restored correctly. https://github.com/JetBrains/intellij-plugins/blob/master/Dart/src/com/jetbrains/lang/dart/lexer/DartLexer.java
I found only one solution which actually seems to restore the correct state. It is the IntelliJ-Adaptor provided by ANTLR 4. However, the solution of ANTLR is also more complicated. I also fear that the implementation of ANTLR might cause a memory leak. https://github.com/antlr/antlr4-intellij-adaptor/blob/master/src/main/java/org/antlr/intellij/adaptor/lexer/ANTLRLexerAdaptor.java
I would like to know how language plugins are supposed to work with this limitation. Most plugins just doesn't seem to restore the correct state. Is that a problem or an intentional compromise? Maybe a "good guess" for the correct highlighting is good enough? Does the syntax highlighter revalidate the whole file anyway in a few seconds when the user stops typing?