Implementing RestartableLexer interface in 2019.2 causes exception on document change


The RestartableLexer interface in 2019.2 was changed but the code using it appears to be incomplete and causes exception on document change.

The cause is the code in LexerEditorHighlighter passing int state to unpackStateFromData while the ShortBasedStorage unpackStateFromData(int) always throws an unsupported operation exception.

I found no classes implementing RestartableLexer in intellij-community master branch. So it appears that this interface can no longer be used to optimize highlighting on small document changes.

Caused by: java.lang.UnsupportedOperationException: Unable to unpack state, state is not stored in ShortBasedStorage

at com.intellij.openapi.editor.ex.util.ShortBasedStorage.unpackStateFromData(

at com.intellij.openapi.editor.ex.util.SegmentArrayWithData.unpackStateFromData(

at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.isInitialState(

at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.documentChanged(


Hey. Is it possible to share an idea what are you trying to achieve?

From what I see you have implemented a RestartableLexer but in your case it uses 


which can't restore the state number from the segment data. Now you should implement


 with you custom storage or just return the storage that stores both token type and state number properly


Actually, I am implementing RestarableLexer interface my plugin lexer as a class inheriting from Lexer base class in the IDE. The RestartableLexer has all state information passed as int.

I can see that the issue is propably caused by replacing the editor highlighter with LexerEditorHighlighter instead of LayeredLexerEditorHighlighter which will use IntBasedStorage for RestartableLexer.


protected SegmentArrayWithData createSegments() {
return new MappingSegments(getLexer() instanceof RestartableLexer ? new IntBasedStorage() : new ShortBasedStorage());

while LexerEditorHighlighter will let the lexer create the storage if it implements DataStorageFactory or use ShortBasedStorage.

protected SegmentArrayWithData createSegments() {
return new SegmentArrayWithData(myLexer instanceof DataStorageFactory ? ((DataStorageFactory)myLexer).createDataStorage() : new ShortBasedStorage());

Any reason why a similar test for implementation of RestartableLexer in LexerEditorHighlighter is not used to select IntBasedStorage for RestartableLexer which does not implement DataStorageFactory?


I can confirm that changing the code in LexerEditorHighlighter to:

protected SegmentArrayWithData createSegments() {
return new SegmentArrayWithData(myLexer instanceof DataStorageFactory ? ((DataStorageFactory)myLexer).createDataStorage()
: (myLexer instanceof RestartableLexer
? new IntBasedStorage()
: new ShortBasedStorage()));

Resolves the issue.

Implementing DataStorageFactory and returning IntBasedStorage also works.

I think that the above change to LexerEditorHightlighter should be applied to the IDE code to allow RestartableLexer to work without needing to implement DataStorageFactory.

The javadoc for RestartableLexer should probably include information about implementing DataStorageFactory. Otherwise it is hard to figure out what is going on without diving deeper into IDE code.



Andrey, I saw the change in LexerEditorHighlighter in this morning's merge of intellij-community master.

Thank you for the quick fix.


Thank you for noticing the problem. I am glad to help you.