Plugin development - how does CLion handle preprocessor token redefinitions?

Created July 12, 2015 19:21

Hi all,

I'm one of the developers of the GLSL plugin, which adds support for a C-like language to IntelliJ/CLion. The language includes a C-like preprocessor, and I'd like the plugin to support understanding code which relies on #define.

My current solution is to use a PsiBuilderAdapter which overrides getTokenType/getTokenText/advance to automatically add phantom tokens to the parse tree which is builder with a PsiBuilder. This partially works but it means I can't ever replace the token text or output ForeignLeafElement tokens like CLion does.

So I figured that a better way to handle this would be to use a DelegateLexer instead. My current implementation of this, based on LookAheadLexer, is here. It generates the tokens correctly (ie. it generates the same tokens CLion does), but something seems to go catastrophically wrong and many exceptions are repeatedly thrown.

So how does CLion handle this? I was hoping a developer would be able to shed some light or point me in the right direction.

Thanks,
Abigail

1 comment

Permanently deleted user

Created July 16, 2015 15:11

Hello, Abigail.

Yes, it's better to handle this with a lexer. As for error I would try to override com.intellij.lexer.LookAheadLexer#lookAhead only. E.g.

 @Override
 protected void lookAhead(Lexer baseLexer) {
    String tokenText = baseLexer.getTokenText();

    super.lookAhead(baseLexer);

    List<IElementType> replacement = definitions.get(tokenText);
    if (replacement != null) {
        for (IElementType tt : replacement) {
            addToken(baseLexer.getTokenStart(), tt);
        }
    }
}

This should add both original token and substitution sequence without messing with internal structures of the base lexer.

Please sign in to leave a comment.