Custom Language Support Token Separation


I am trying to implement language support plugin for a basic language. I am following the simple language tutorial and on the side I have rust plugin for reference. However the complexity gap between them is huge so some questions are hard to find answers to from both sources.

Here is my question: What is the best way to allow spaces between tokens which DO NOT require spaces, however force spacing between tokens which do?

i.e. class Foo{ <- here first space is mandatory, but the second one (before '{' symbol) can be omitted.

I know that I can control it by manipulating states from .flex file. but seems a lot of unnecessary work, which should have a better solution anyway

Comment actions Permalink


Another solution would be to handle whitespaces "normally" in lexer and annotate the lack of them in the required places as errors with an Annotator implementation (e.g., "Whitespace expected, got '{'" in your example "class Foo{"). You would need a definition of which pairs of tokens require spaces between and annotate based on this definition. This definition could also be reused in ParserDefinition.findSpaceRequirement(), which is used for inserting spaces during AST modifications.

Comment actions Permalink

Karol Lewandowski

Thank you for the response, to clarify:

by "normally" you mean just return whitespace token when lexer encounters whitespace and then define space requiring tokens with a definition and show errors when they are not satisfied using annotator?

Comment actions Permalink

Yes, this is exactly what I meant.


Please sign in to leave a comment.