Plugin development - how does CLion handle preprocessor token redefinitions?

Hi all,

I'm one of the developers of the GLSL plugin, which adds support for a C-like language to IntelliJ/CLion. The language includes a C-like preprocessor, and I'd like the plugin to support understanding code which relies on #define.

My current solution is to use a PsiBuilderAdapter which overrides getTokenType/getTokenText/advance to automatically add phantom tokens to the parse tree which is builder with a PsiBuilder. This partially works but it means I can't ever replace the token text or output ForeignLeafElement tokens like CLion does.

So I figured that a better way to handle this would be to use a DelegateLexer instead. My current implementation of this, based on LookAheadLexer, is here. It generates the tokens correctly (ie. it generates the same tokens CLion does), but something seems to go catastrophically wrong and many exceptions are repeatedly thrown.

So how does CLion handle this? I was hoping a developer would be able to shed some light or point me in the right direction.


1 comment
Dmitry Semeniouta
Comment actions Permalink

Hello, Abigail.

Yes, it's better to handle this with a lexer. As for error I would try to override com.intellij.lexer.LookAheadLexer#lookAhead only. E.g.

protected void lookAhead(Lexer baseLexer) {
    String tokenText = baseLexer.getTokenText();


List<IElementType> replacement = definitions.get(tokenText);
(replacement != null) {
        for (IElementType tt : replacement) {
            addToken(baseLexer.getTokenStart(), tt);

This should add both original token and substitution sequence without messing with internal structures of the base lexer.


Please sign in to leave a comment.