I have followed the custom language plugin tutorial and implemented a FlexLexer and parser which works fine. I've added the SyntaxHighlighting which also works great however, once I change the text of the document I get the following exception: 

com.intellij.openapi.editor.ex.util.LexerEditorHighlighter$InvalidStateException: com.intellij.openapi.editor.ex.util.LexerEditorHighlighter(com.misset.opp.omt.OMTLexerAdapter): Error updating after DocumentEventImpl[myOffset=999, myOldLength=1, myNewLength=0, myOldString='d', myNewString=''].
at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.documentChanged(LexerEditorHighlighter.java:338)
at com.intellij.openapi.editor.impl.DocumentImpl.changedUpdate(DocumentImpl.java:892)


Lexer is not progressing after calling advance()


After this all highlighting is out of sync. Reopening the document fixes it. 

I read some things about needing a restartable lexer but couldn't really find anything about that. Is there somewhere an example where this is used?


Kr, Tim


I managed to figure out what was going on. The LexerEditorHighlighter  has a loop detection that checks if the lexer is progressing on state, position and token. In my DSL I capture indentation levels by tracking the whitespace and when the first non-whitespace character comes up, returns the Indent/Dedent tokens. While it has to return more tokens it will pushback the non-whitespace character.

This mechanisme will cause the LexerEditorHighlighter to throw the error once identantion levels > 1. 

I could fix it by having the lexer return alternative INDENT/INDENT2 and DEDENT/DENENT2 tokens, however, I wonder if there is a better way to deal with these kind of processes. I can imagine having the INDENT levels being returned as soon as they are discovered, but for the DEDENT I don't have a clue, since there is no actual whitespace to increase the token position on, it reports the lack of whitespace.

Any suggestions on how to improve this mechanisme?


Hello Tim, my DSL is also indention based and I also encountered the same problem.  Here's what I've found in intellij idea CE, hope that'll help:

the official python plugin just implement 2 lexer separately for highlight/parsing. Only when doing highlighting it will check if  the lexer is progressing, so we can just ignore and do not return the empty tokens when doing advance(). And for parsing, all empty/not empty tokens need to be returned to build the correct parsing tree.

source code from IntelliJ CE:

the lexer for parsing: https://github.com/JetBrains/intellij-community/blob/master/python/python-psi-impl/src/com/jetbrains/python/lexer/PythonIndentingLexer.java

the lexer for highlighting: https://github.com/JetBrains/intellij-community/blob/master/python/python-psi-impl/src/com/jetbrains/python/lexer/PythonHighlightingLexer.java

Here's how these lexers registered:

register for parsing: https://github.com/JetBrains/intellij-community/blob/master/python/python-psi-impl/src/com/jetbrains/python/PythonParserDefinition.java#L33

register for highlighting: https://github.com/JetBrains/intellij-community/blob/master/python/src/com/jetbrains/python/highlighting/PyHighlighter.java#L64



Hi Xiarui,

thanks for taking the time to leave a comment. I'll have a look at the suggestion, looks promising.

Cheers, Tim


Hey! I'm currently experiencing the exact same problem with indentation levels.
I've taken a look at Xiarui Xr ‘s propsed solution, however I’m looking for a solution that utilizes `.bnf` and `.flex` files (instead of writing the Java code myself).

I can't seem to make both parser and lexer happy at the same time, so it seems.


Please sign in to leave a comment.