Support for preprocessing in custom language

Answered

Permanently deleted user

Created February 24, 2020 15:01

I am going to implement plugin for custom language X that supports C-like preprocessing (#define, #ifdef ..)

What would be the easiest solution to parse code in such language?

As macros could be called in almost every place and may resolve to any sequence of tokens, defining a common grammar for language X + preprocessor seems to be overcomplicated.

Is there any description how it is implemented in CLion for C/C++ or any other language that supports preprocessing?

4 comments

Jakub Chrzanowski

Created February 25, 2020 04:55

Hi Karol,

Unfortunately, there is no such documentation or example for the C/C++ implementation available.

For now, I'd suggest you review the other Language-like plugins in our repository for some inspiration: https://plugins.jetbrains.com/search?headline=48-languages&products=idea&tags=Languages

Most of them are OpenSource projects hosted in GitHub - it may be a good starting point for you together with the Custom Language Support docs as well.

Yann Cebron

Created October 12, 2020 08:25

Update: https://intellij-support.jetbrains.com/hc/en-us/community/posts/360009710039-support-for-include-file-ext-directive?page=1#community_comment_360002275160

Permanently deleted user

Created October 12, 2020 22:25

Thank you for your answer.

2 follow up questions:

Question #1

Grammar-Kit has little to do with all the above as we keep lexer and parser layers clearly separated

…

// preprocessor rules can be defined in the same Grammar-Kit grammar

I agree with it in principle but I see one issue.

Your solution assumes that it is possible to pass MyBuilder instance to the generated parser.

Please check my issue that I have raised here: https://github.com/JetBrains/Grammar-Kit/issues/241

Several methods in GeneratedParserUtilBase class downcast PsiBuilder to PsiBuilderImpl which will not work with MyBuilder.

I cannot also workaround it by inheriting from PsiBuilderImpl because eof() method is final there.

Question #2

we keep lexer and parser layers clearly separated.

Conditionals like #ifdef, #if mean that in some cases fragments of code are “disabled”
Macros may have formal parameters that should be substituted with actual that may contain different characters including commas as long as they conform to macro call grammar.
To decide whether code should be disabled (commented?) we need to evaluate the expression provided to #if directive.

If I understood correctly all those steps should happen in Lexer. Am I right? I am ok if yes but I just would like to confirm that I understand it correctly.

Yann Cebron

Created July 12, 2021 14:17

Several methods in GeneratedParserUtilBase class downcast PsiBuilder to PsiBuilderImpl which will not work with MyBuilder.

FTR https://github.com/JetBrains/Grammar-Kit/issues/241#issuecomment-752980474 - You can copy it to your project and customize to your requirements.

If I understood correctly all those steps should happen in Lexer. Am I right?

Correct

Please sign in to leave a comment.