JFlex lexer of custom language integration keeps outputing EOF. Why?

Hi,

I've written a JFlex lexer definition for a custom language integration. Because I need the EOF token in the parser, the flex-file ends with:
<<EOF>>  { return RTypes.R_EOF; }

However, when running it, the generated Lexer goes into an infinite loop which initially lexes the input correctly but then continues to output just EOF tokens forever. Example

SYMBOL
<-
SYMBOL
(
STR_CONST
,
SYMBOL
=
STR_CONST
)
;
EOF
EOF
EOF
EOF
EOF
EOF
...



What's wrong with my approach? What is the right place in the flex file to match the end of the file? The java.flex in the idea sources seems to do it in the same way.

Best,
Holger Brandl
3 comments
Comment actions Permalink

So let me rephrase the questions: Does anyone of you know a working Jflex-based lexer of an intellij custom language which creates an EOF token?

I've followed the jflex documentation by adding the following line to the end of my flex-file:
<<EOF>>  { return RTypes.R_EOF; }
However this gives the above-described error behavior when running it using the FlexAdapter from Intellij.

Any help would be highly appreciated!

Best,
Holger Brandl

0
Comment actions Permalink

Why do you need an explicit EOF token? EOF is signified by returning null from getTokenType().

0
Comment actions Permalink

Hello Dimitry,

I'm using the GrammarKit for my plugin: I specify the grammar of my language as a bnf-file. By regenerating the parser from the bnf automatically, it's very convenient to improve the plugin.

As my language grammar allows for linebreaks just after some grammar elements I need the EOF token (as alternative to EOL) in some parsing rules (to match the last statement in a file). So convicing the lexer to return a proper token for the end of the file would save me from messing around in the generated parser code.

Best,
Holger

0

Please sign in to leave a comment.