Custom Language Plugin Development

Are there any simple custom language plugin examples/tutorials around. One that explains more than just "use JFlex"...

I often have to work with custom languages, some simple, some complex, and I usually have an ANTLR grammar definition around, or can put one together. Converting this knowledge in to a plugin does not seem easy right now.

After reading the custom language plugin doc on the IntelliJ website, I was following along nicely, up to the point of "go use JFlex".  Darn. I had that whole language file type registration down along with its pretty little icon.

Yes, I've seen the monster Javascript plugin source code (and the Clojure one).  It would be *awesome* if there was a simple, complete,  IntelliJ project that defined a simple 5-10 keyword grammar with numerics (+ and - ops) that I could open up, compile (Ctrl-F9) and then run the plugin that used the aforementioned JFlex (and its provided Adapter class).
It would be really nice if I didnt have to wait 4 hours (ok, a slight stretch of reality, but it sure feels like it) for a fresh new instance of IDEA to start up just to test my plugin - why cant I load this in to my current running editor?
With a simple, complete plugin project I could see the basic syntax highlight, variable definition, and some of the simpler validations and operations, and how to go about changing them. If the project requires an Ant script to do the JFlex integration, so be it, just add it in there.

I'm truly amazed that anyone outside of Jetbrains have developed a custom language plugin given the docs and examples around.

With that said, the file type syntax highlight options (in Settings-->File Types) are very nice for simple highlighting, so thanks for that.  However, I'd like to progress to syntax validation and variable definition lookup etc.

8 comments
Comment actions Permalink

Hello, Nick.
I totally agree that javascript plugin code is bloated with some complex logic. The javascript language itself is pretty complex.
Try looking at nginx plugin ( http://plugins.intellij.net/plugin/?id=4411 ). Lexer is pretty simple there and parser is simple as well. Some basic  Some annotator routines are implemented as well.
Don't forget to install PsiViewer plugin to view resulting PSI.
Source code for nginx plugin is available on http://code.google.com/p/idea-nginx/
My advice is to disable all plugins for sandboxed IDEA instance and creating as empty project as you can. That can minimize the harm of restarting IDEA on substantial changes. Don't forget to use hot swap if your code changes are subtle (e.g. editing method body only).
One needs patience and some Sherlock Holmes skills to develop plugins for IntelliJ . I wish Jetbrains paid more attention to documenting/opensourcing idea classes.

0
Comment actions Permalink

I agree, some more documentation around this would be great, it's currently really difficult to develop a language plugin and it's a shame, because it's possible to do such awesome stuff with it. I've just started working on one for Scheme, based on the Clojure one. I was pretty lucky because Scheme is so close to Clojure, so a couple of days ripping stuff out and it's come out pretty nicely. But particularly implementing the whole parser and PSI by hand is prohibitive for a lot of people I think, it's a huge amount of work. I think language support is one of the areas where IntelliJ could really stand out from Eclipse and Netbeans, but it's so difficult I don't think many people bother. I'm working on mine in my (limited) spare time, if/when it gets frustrating I'm not sure how far I'll be able to take it, realistically. I think small examples like the nginix one are essential for people getting started, IMO Jetbrains should take a couple and clean them up as examples of how to get started.

BTW does the PsiViewer plugin still work in IntelliJ 8? I didn't even install it since I saw it hadn't been updated since v5 and saw some comments on the plugin that people had problems with it. If so, I'll give it a go.

Cheers,
Colin

0
Comment actions Permalink

Well, I'd say that parser implementation is the easiest part of writing a really cool language plugin. It's hard to start, but as soon as you get the point, you write parsing code automatically. The problem is it's hard to get the point right for the first time.
The hardest part for me is formatter stuff - i haven't understood yet how these funny blocks/indents/wraps etc work.
PsiViewer does work in 8 (didn't check 9) and it's a miracle. In fact, I don't know how to debug parser without it. Maybe with some println stuff, but that doesn't deliver much fun.
Sometimes I think some plugin writers have at their disposal all IDEA's source code. I have none, but i got intellijad instead )

0
Comment actions Permalink

Many thanks for the tips.  Are you still involved with the JFlex plugin for IntelliJ? I know the API has changed in Maia, but it would be nice if... :-)

Ill try and develop a simple language plugin and write a tutorial on it, then enhance the "language" once I get to grips with things.


However, my point about not being able to load a plugin in to a running IDEA instance still stands - the code/test cycle is painfully slow.  Im still amazed to think what some plugin authors must have endured to get their plugins working and as functional as they are.

Nick

0
Comment actions Permalink

Ok, I'll try to fix Maia issues for JFLex plugin.
However, you can still develop plugins for Maia in Diana, just use the appropriate SDK.

0
Comment actions Permalink

BTW see the Clojure plugin for a way to test parsers outside the IDE. I haven't had time to fully understand the code yet, but copying it and substituting my own tests works nicely.

The PsiViewer plugin is indeed awesome :-)

0
Comment actions Permalink

Im just going to repeat this in the slim chance that anyone at Jetbrains missed this point - the documentation/SDK for writing a custom language plugin in IntelliJ is pitiful.

We have all this potential power and ability in IDEA, but you make it almost impossible to use and gain serious benefit from.  The same comment can be directed at Structural Search and Replace, but I'll leave that for a different day.

These are one of the most powerful features of the editor, give some examples and a step by step tutorial:

1.) Here are the basics of the language plugin process- you have that covered in the opening paragraph.
2.) A tutorial on writing a highlighter for a simple language (a couple of keywords, basic arithmetic on ints and floats, curly braces and parenthesis), along with all the supporting classes and implementations of the lexer/parser/whatever
3.) Move on to brace matching, code folding
4.) Error highlighting - the code as written is just syntactically wrong, so here's how you make IDEA do this...
5.) Writing a simple intention - for example, if the language uses := for assignment, and the developer wrote a = 6; then write a tutorial on how to make this work
6.) Embedded languages - how do you make your custom language useable from within a Java program - similar to how SQL / HQL works, so that the various error highlighting etc all works.

The "language" doesnt have to do anything - just show how to get this up and running, see the keyword highlighting, code folding, error analysis etc. Being able to open a file in IDEA, have it recognized, show the syntax highlights, and basic syntax error analysis would be awesome.  Cover points 1-4.  Intentions, and all the rest are just icing on the cake.

Taking any of the existing plugins, and trying to pare them down to a simplified case not workable.  Ive spent many hours now trying to strip down the JavaScript plugin to something useable, and Im lost in the maze of  lexers, parsers and support classes, and stuff just isnt working.  Learning this would be much better from a simple example, buildling up to a more complex one.  Taking a fully implement language plugin with all the bells and whistles might be good as a reference for an experienced language plugin developer, but its useless for a beginner.

I fully appreciate that this is a very complex task.  But knowing this fact, please provide some better examples and docs/tutorials for the complex and very powerful parts of the IDE.  There's no reason why we couldn't use IDEA for any language we came across - that certainly seems to have been a significant push over the last few releases, but with the current state of the single tutorial/document to assist anyone thinking about developing a language plugin, Jetbrains is missing out on a huge boost to the language support.

0

Please sign in to leave a comment.