Implementing RestartableLexer interface in 2019.2 causes exception on document change
已回答
The RestartableLexer interface in 2019.2 was changed but the code using it appears to be incomplete and causes exception on document change.
The cause is the code in LexerEditorHighlighter passing int state to unpackStateFromData while the ShortBasedStorage unpackStateFromData(int) always throws an unsupported operation exception.
I found no classes implementing RestartableLexer in intellij-community master branch. So it appears that this interface can no longer be used to optimize highlighting on small document changes.
Caused by: java.lang.UnsupportedOperationException: Unable to unpack state, state is not stored in ShortBasedStorage
at com.intellij.openapi.editor.ex.util.ShortBasedStorage.unpackStateFromData(ShortBasedStorage.java:66)
at com.intellij.openapi.editor.ex.util.SegmentArrayWithData.unpackStateFromData(SegmentArrayWithData.java:115)
at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.isInitialState(LexerEditorHighlighter.java:132)
at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.documentChanged(LexerEditorHighlighter.java:167)
请先登录再写评论。
Hey. Is it possible to share an idea what are you trying to achieve?
From what I see you have implemented a RestartableLexer but in your case it uses
which can't restore the state number from the segment data. Now you should implement
with you custom storage or just return the storage that stores both token type and state number properly
Actually, I am implementing RestarableLexer interface my plugin lexer as a class inheriting from Lexer base class in the IDE. The RestartableLexer has all state information passed as int.
I can see that the issue is propably caused by replacing the editor highlighter with LexerEditorHighlighter instead of LayeredLexerEditorHighlighter which will use IntBasedStorage for RestartableLexer.
while LexerEditorHighlighter will let the lexer create the storage if it implements DataStorageFactory or use ShortBasedStorage.
Any reason why a similar test for implementation of RestartableLexer in LexerEditorHighlighter is not used to select IntBasedStorage for RestartableLexer which does not implement DataStorageFactory?
I can confirm that changing the code in LexerEditorHighlighter to:
Resolves the issue.
Implementing DataStorageFactory and returning IntBasedStorage also works.
I think that the above change to LexerEditorHightlighter should be applied to the IDE code to allow RestartableLexer to work without needing to implement DataStorageFactory.
The javadoc for RestartableLexer should probably include information about implementing DataStorageFactory. Otherwise it is hard to figure out what is going on without diving deeper into IDE code.
Andrey, I saw the change in LexerEditorHighlighter in this morning's merge of intellij-community master.
Thank you for the quick fix.
Thank you for noticing the problem. I am glad to help you.