Language-API: JFlex lexer with lookahead is causing trouble

Hi Maxim & Co,

if I create a JFlex lexer that's using lookahead rules like

"comment" / ? "("

the resulting lexer doesn't work properly in IDEA and is causing all kinds of weird
effects, such as tokens with a text range of are reported to be at , strange
highlighting exceptions all over the place, etc.

I got this fixed by changing the skeleton file that's included in the dev-package from

public final int getTokenStart(){
return zzStartRead - zzPushbackPos;
}

to just

public final int getTokenStart(){
return zzStartRead;
}

Not sure though if this will cause other weird effects, but I haven't seen any so far.

Am I doing anything wrong, or is this a bug?

Sascha

3 comments
Comment actions Permalink

Any idea about that? Can you safely add that modification to the skeleton file that is
distributed with the dev-package? IIRC, zzPushbackPos is always zero anyway unless the
lexer uses lookahead rules (JFlex generates different code), so it shouldn't break anything.

While you're at it, could you please also add

public void setCharAt(boolean b) {
Options.char_at = b;
}

to JFlex/anttask/JFlexTask.java for easier usage from Ant-builds?

Oh, and including the manifest from the JFlex sources in the jar would be nice as well
(that way the jar can be run with java -jar JFlex.jar ...) ;)

Thanks,
Sascha

0
Comment actions Permalink

Yes, please fix the mainfest.

Sascha Weinreuter schrieb:

Any idea about that? Can you safely add that modification to the skeleton file that is
distributed with the dev-package? IIRC, zzPushbackPos is always zero anyway unless the
lexer uses lookahead rules (JFlex generates different code), so it shouldn't break anything.

While you're at it, could you please also add

public void setCharAt(boolean b) {
Options.char_at = b;
}

to JFlex/anttask/JFlexTask.java for easier usage from Ant-builds?

Oh, and including the manifest from the JFlex sources in the jar would be nice as well
(that way the jar can be run with java -jar JFlex.jar ...) ;)

Thanks,
Sascha

0
Comment actions Permalink

Hello Sven,

All of these issues will be fixed in the next build of IDEA.

Thanks for your feedback!

SS> Yes, please fix the mainfest.
SS>
SS> Sascha Weinreuter schrieb:
SS>
>> Any idea about that? Can you safely add that modification to the
>> skeleton file that is distributed with the dev-package? IIRC,
>> zzPushbackPos is always zero anyway unless the lexer uses lookahead
>> rules (JFlex generates different code), so it shouldn't break
>> anything.
>>
>> While you're at it, could you please also add
>>
>> public void setCharAt(boolean b) {
>> Options.char_at = b;
>> }
>> to JFlex/anttask/JFlexTask.java for easier usage from Ant-builds?
>>
>> Oh, and including the manifest from the JFlex sources in the jar
>> would be nice as well (that way the jar can be run with java -jar
>> JFlex.jar ...) ;)

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0

Please sign in to leave a comment.