I am wandering about different types of lexers.
They all implement the lexer interface. I tried to make my own lexer that would return whole text as one token but i'm getting Out of memory exception - i suppose because of some value i return. So the first question here: what's the scheme of lexer usage. Which return values are being passed to lexer back?