I'm writing custom language plugin and my language uses Python-like indentation for blocks.
I use LookAheadLexer to handle indentation.
The problem is that lexer tokens should cover some chunk of source code. For example, if I have:
There are spaces to be associated with INDENT tokens. Like: line1, CRLF, INDENT, line2, CRLF, INDENT, INDENT, line3, CRLF, line4.
But there is no room to associate DEDENT tokens with, and PSI structure I actually need is line1, CRLF, INDENT, line2, CRLF, INDENT, INDENT, line3, CRLF, DEDENT, DEDENT, line4.
When I try to use addToken(DEDENT) in my lookAhead method it doesn't work (I either try to cover an empty range of text or some range that is already covered by another token).
How should INDENTs and DEDENTs be handled exactly?