Freemarker plugin - Flex or JavaCC, that is the question

Hi all,

I'm trying to write a plugin to add Freemarker support to IntelliJ. Now I'd like to understand how to write a lexer for syntax highlighting. My issue is that if I look at Freemarker's source, their parser is written for JavaCC, not JFlex. Knowing that I haven't written a parser in a long time, I'm trying to find the best approach.
- is there a way to convert a .jj file into a .flex one
- is it possible to implement a lexer in JavaCC instead of JFlex (and are there examples of plugins doing so)?
- should I write a Flex parser from scratch?

What do you think?

8 comments
Comment actions Permalink

Hi!

As far as i know javaCC is both lexer and parser while jflex is only a lexer.
I know of one plugin (GroovyJ) that used ANTLR as lexer and parser, but i think the typical approach for custom languages in IDEA is to write lexer and parser manually (lexer is jflex-generated and parser is completely by hand) since you have to continue your parsing whatever user types in.

0
Comment actions Permalink

Really? GroovyJ used ANTLR for the parser? Can the GroovyJ guys comment on how it worked out? I'll have to get the code and take a look. IMO hand-written parsers is one of the biggest blocks to writing new language plugins.

Cheers,
Colin

0
Comment actions Permalink

Hi, Colin!

Yes i'm sure they did it, but they commented then:

+Alas, this approach to adding Groovy support to IntelliJ IDEA wasn't the correct one. A
hand-written fault-tolerant parser is what should be implemented instead.+

Unfortunately i lost the sources of GroovyJ plugin and could not find it now. However here http://plugins.intellij.net/plugin/?id=260 the plugin is marked as opensource, so i think you can contact the author with this.

Hand-written parsers are a real trouble for custom languages but i'm afraid it's the only way to implement a good language support.

Good luck,
jay

0
Comment actions Permalink

Ok, thanks for the info! It's a shame there's still no good way of generating a parser based on a grammar. This would be a sure fire winner for the plugin contest...

Cheers,
Colin

0
Comment actions Permalink

It's not that hard. Just try it.


Colin Fleming wrote:

Ok, thanks for the info! It's a shame there's still no good way of generating a parser based on a grammar. This would be a sure fire winner for the plugin contest...

Cheers,
Colin

0
Comment actions Permalink

Is that based on experience? I'd be interested to know why the Groovy folk thought it was the wrong way to go.

0
Comment actions Permalink

Colin Fleming wrote:

Is that based on experience? I'd be interested to know why the Groovy folk thought it was the wrong way to go.

Yes. I was experimenting with it for several weeks. In my opinion it's
the simplest language API available today. I'm not saying it's easy though.

Groovy folk already had antlr grammar from Groovy language and he just
added it to plugin. I don't think he thought PSI parser is a wrong way
to go. Just take a look at groovy grammar, it's just huge. No wonder he
tried to avoid reimplementing it in Java.

Language construction never been easy, it doesn't matter if you do it by
hand or by Antlr.

-sergiy

0
Comment actions Permalink

In case anyone's interested, I managed to get a copy of the source here:

http://svn.groovy.codehaus.org/browse/groovy/trunk/groovy/ide/groovyj

Thanks for the comments, Sergiy, I'll take a look at using ANTLR. I agree it's never easy but ANTLR would sure help.

Cheers,
Colin

0

Please sign in to leave a comment.