How to implement Lexer.start/FlexLexer.reset when the lexer carries a stack of states

Here is my current implementation which I might end up using. It actually fixed some strange behavior I have noticed in syntax highlighting. Not sure if this is a good solution, though.

I noticed while debugging that IDEA did never call reset with `initialState != 0`. Maybe IDEA only resets the Lexer to positions where Lexer.getState() has returned 0?

User code in `*.flex` file:

private final TLongArrayList states = new TLongArrayList();
private final TLongIntHashMap stateIndexMap = new TLongIntHashMap();

{
  states.add(YYINITIAL);
  stateIndexMap.put(YYINITIAL, 0);
}

private int currentStateIndex = 0;
private int parentStateIndex = 0;

public int getStateIndex() {
  return currentStateIndex;
}

public void restoreState(int stateIndex) {
  long state = states.get(stateIndex);
  currentStateIndex = stateIndex;
  parentStateIndex = (int) (state >> 32);
  yybegin((int) state);
}

private void pushState(int yystate) {
  long state = ((long) currentStateIndex << 32) | ((long) yystate & 0x0FFFFFFFFL);
  int stateIndex = stateIndexMap.get(state); // Returns 0 if not found
  if (stateIndex == 0 && state != YYINITIAL) {
    stateIndex = states.size();
    states.add(state);
    stateIndexMap.put(state, stateIndex);
  }
  parentStateIndex = currentStateIndex;
  currentStateIndex = stateIndex;
  yybegin(yystate);
}

private void popState() {
  restoreState(parentStateIndex);
}

Adapter implementation:

public class MyLexer extends FlexAdapter {
    public MyLexer() {
        super(new FlexLexer() {
            private final _MyLexer lexer = new _MyLexer(null);

            @Override
            public void yybegin(int state) {
                lexer.restoreState(state);
            }

            @Override
            public int yystate() {
                return lexer.getStateIndex();
            }

            @Override
            public int getTokenStart() {
                return lexer.getTokenStart();
            }

            @Override
            public int getTokenEnd() {
                return lexer.getTokenEnd();
            }

            @Override
            public IElementType advance() throws IOException {
                return lexer.advance();
            }

            @Override
            public void reset(CharSequence buf, int start, int end, int initialState) {
                lexer.reset(buf, start, end, initialState);
                lexer.restoreState(initialState);
            }
        });
    }
}

I found this commit in LexerEditorHighlighter which mentions that you can use LexerTestCase.checkCorrectRestart to test if your lexer is working correctly. The comment also mentions LayeredLexer, but I am not sure why and when it should be used.

Note that IntelliJ actually does not distinguish between different non-initial states by default. IntelliJ will only restart at tokens which have the initial state. You could therefore avoid my complex implementation from above and just return "stack.size()" as state, assuming your stack is initially empty. However, if your lexer does not return to its initial state for a long time, this can cause a sluggish user experience as described in the linked comment.

If you want to avoid such sluggish user experience, it seems you must implement a proper state handling as demonstrated in my last comment. Your lexer must then implement RestartableLexer (source) which tells IntellJ that the lexer can be restarted at other states as well. Note that you should use LexerTestCase.checkCorrectRestartOnEveryToken instead of LexerTestCase.checkCorrectRestart for testing such lexer. Unfortunately, this interface has experimental methods without default implementations, which means your plugin will break if JetBrains changes their experimental API of this interface.

Late reply, but hopefully still helpful.

https://github.com/JetBrains/intellij-community/blob/master/plugins/sh/core/src/com/intellij/sh/lexer/Sh.flex is a good example of state management with a Flex-based lexer. You basically use JFlex's `int` state as state, which is returned as the IDE lexer's state. I don't see a reason to keep states as `long` in the current implementation of the Nix lexer.

Unless you're implementing a complex lexer with `RestartableLexer`, just return `0` to make a lexer restartable (`YYINITIAL` is `0`).

Joachim Ansorg, thanks for your reply. As long as you are not stacking your states, you are right. When you are not using `RestartableLexer`, you may even use a stack as long as you are not calling `pop` while in state `YYINITIAL`.

In my case, I took the previous Flex state from the stack whenever I was reading `}`. Just the state integer of Flex wasn't enough, I had to encode the stack into the overall state of my lexer. Anyway, I could have avoided the complexity by just not using `YYINITIAL` as long as my stack is not empty (assuming I am not implementing `RestartableLexer`).