Trailing whitespace in source file kills my SyntaxHighlighter (in a new lang plugin)
Answered
Hello,
I did some initial work on a language plugin and as far as lexer and parser go, it works well. But once I activate SyntaxHighlighter, it fails on files where I have a newline after the last scanned token. It happens here:
java.lang.IllegalStateException: Unexpected termination offset for lexer FlexAdapter for org.m3w.m3.Modula3Lexer
at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.doSetText(LexerEditorHighlighter.java:467)
When I remove trailing whitespace and reopen a file, it does not break. On line 467 I see my last token (segment) ends one byte short of textLength.
I am sure I am missing something in definitions of lexer or parser... Any hints?
Thanks,
dd
Please sign in to leave a comment.
Ok, I solved this.
Just for everybody else falling into a trap like this - it is not a good idea to ignore whitespace in .flex, much better is to return TokenType.WHITE_SPACE instead.
Thanks for pointing out this explanation Dragiša, I had the same problem. In my case, it was ONLY triggered when I was missing a newline at the end of a file.
I solved it by adding a rule for <<EOF>> for the state that it was in when reaching the end of the file, and made it end the token when doing so.