Implementing RestartableLexer interface in 2019.2 causes exception on document change

Answered

The RestartableLexer interface in 2019.2 was changed but the code using it appears to be incomplete and causes exception on document change.

The cause is the code in LexerEditorHighlighter passing int state to unpackStateFromData while the ShortBasedStorage unpackStateFromData(int) always throws an unsupported operation exception.

I found no classes implementing RestartableLexer in intellij-community master branch. So it appears that this interface can no longer be used to optimize highlighting on small document changes.

Caused by: java.lang.UnsupportedOperationException: Unable to unpack state, state is not stored in ShortBasedStorage

at com.intellij.openapi.editor.ex.util.ShortBasedStorage.unpackStateFromData(ShortBasedStorage.java:66)

at com.intellij.openapi.editor.ex.util.SegmentArrayWithData.unpackStateFromData(SegmentArrayWithData.java:115)

at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.isInitialState(LexerEditorHighlighter.java:132)

at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.documentChanged(LexerEditorHighlighter.java:167)


5 comments
Comment actions Permalink

Hey. Is it possible to share an idea what are you trying to achieve?

From what I see you have implemented a RestartableLexer but in your case it uses 

com.intellij.openapi.editor.ex.util.ShortBasedStorage

which can't restore the state number from the segment data. Now you should implement

com.intellij.openapi.editor.ex.util.DataStorageFactory

 with you custom storage or just return the storage that stores both token type and state number properly

com.intellij.openapi.editor.ex.util.IntBasedStorage
0
Comment actions Permalink

Actually, I am implementing RestarableLexer interface my plugin lexer as a class inheriting from Lexer base class in the IDE. The RestartableLexer has all state information passed as int.

I can see that the issue is propably caused by replacing the editor highlighter with LexerEditorHighlighter instead of LayeredLexerEditorHighlighter which will use IntBasedStorage for RestartableLexer.

 

  @NotNull
@Override
protected SegmentArrayWithData createSegments() {
return new MappingSegments(getLexer() instanceof RestartableLexer ? new IntBasedStorage() : new ShortBasedStorage());
}

while LexerEditorHighlighter will let the lexer create the storage if it implements DataStorageFactory or use ShortBasedStorage.

  @NotNull
protected SegmentArrayWithData createSegments() {
return new SegmentArrayWithData(myLexer instanceof DataStorageFactory ? ((DataStorageFactory)myLexer).createDataStorage() : new ShortBasedStorage());
}

Any reason why a similar test for implementation of RestartableLexer in LexerEditorHighlighter is not used to select IntBasedStorage for RestartableLexer which does not implement DataStorageFactory?

0
Comment actions Permalink

I can confirm that changing the code in LexerEditorHighlighter to:

  @NotNull
protected SegmentArrayWithData createSegments() {
return new SegmentArrayWithData(myLexer instanceof DataStorageFactory ? ((DataStorageFactory)myLexer).createDataStorage()
: (myLexer instanceof RestartableLexer
? new IntBasedStorage()
: new ShortBasedStorage()));
}

Resolves the issue.

Implementing DataStorageFactory and returning IntBasedStorage also works.

I think that the above change to LexerEditorHightlighter should be applied to the IDE code to allow RestartableLexer to work without needing to implement DataStorageFactory.

The javadoc for RestartableLexer should probably include information about implementing DataStorageFactory. Otherwise it is hard to figure out what is going on without diving deeper into IDE code.

 

0
Comment actions Permalink

Andrey, I saw the change in LexerEditorHighlighter in this morning's merge of intellij-community master.

Thank you for the quick fix.

0
Comment actions Permalink

Thank you for noticing the problem. I am glad to help you.

0

Please sign in to leave a comment.