How to host a language?

Is there an example available that shows how to embed one language into another. I have tried embedding both Java and Javascript into my language and it failed with different errors each time.

Do I need to use IChameleonElement at all, or just IChameleonElementType? When do I need to override PsiFile.getPsiRoots()? Could you please give me an overview of how an implementation of getPsiRoots should work?

My language looks like this:

-


someString =
"and so is this"


this is not code

x = "more javascript code here"
/* this is the start of the comment


this is not inside the comment

this is the end of the comment */

-



I have already created a lexer/parser/parser definition/highlighter for the host language (lines beginning with ">" are code, every other line is a comment) that works correctly. In its lexer, I the segments after ">" are returned as single IChameleonElementType tokens (one on each line), with the language as Javascript.

When I look at my file with PSIViewer, I can see that the Javascript code is getting parsed because there are Javascript PSI elements in my code. But, this happens only for very short Javascript fragments, and only when I do not type "too slow." For example, if I type "asdf" really quickly it will get parsed as a Javascript expression. But, if I type "a" then "s" slowly, then when I type the "s" I get the exception below.

Could you please give me some hints as to what might cause this exception?

Thanks,
Brian

java.lang.NullPointerException
at com.intellij.psi.impl.source.tree.ChangeUtil.a(ChangeUtil.java:279)
at com.intellij.psi.impl.source.tree.ChangeUtil.replaceAllChildren(ChangeUtil.java:143)
at com.intellij.psi.impl.source.text.BlockSupportImpl.reparseRangeInternal(BlockSupportImpl.java:91)
at com.intellij.psi.impl.source.text.BlockSupportImpl.reparseRange(BlockSupportImpl.java:5)
at com.intellij.psi.impl.PsiDocumentManagerImpl.commit(PsiDocumentManagerImpl.java:131)
at com.intellij.psi.impl.PsiDocumentManagerImpl$1.run(PsiDocumentManagerImpl.java:5)
at com.intellij.openapi.application.impl.ApplicationImpl.runWriteAction(ApplicationImpl.java:381)
at com.intellij.psi.impl.PsiDocumentManagerImpl.commitDocument(PsiDocumentManagerImpl.java:93)
at com.intellij.psi.impl.PsiDocumentManagerImpl.commitAllDocuments(PsiDocumentManagerImpl.java:29)
at com.intellij.codeInsight.daemon.impl.TextEditorBackgroundHighlighter.a(TextEditorBackgroundHighlighter.java:10)
at com.intellij.codeInsight.daemon.impl.TextEditorBackgroundHighlighter.createPassesForEditor(TextEditorBackgroundHighlighter.java:22)
at com.intellij.codeInsight.daemon.impl.TextEditorBackgroundHighlighter.createPassesForEditor(TextEditorBackgroundHighlighter.java:24)
at com.intellij.codeInsight.daemon.impl.DaemonCodeAnalyzerImpl.a(DaemonCodeAnalyzerImpl.java:75)
at com.intellij.codeInsight.daemon.impl.DaemonCodeAnalyzerImpl.access$2000(DaemonCodeAnalyzerImpl.java:1)
at com.intellij.codeInsight.daemon.impl.DaemonCodeAnalyzerImpl$8$1UpdateEditorRunnable.run(DaemonCodeAnalyzerImpl.java:1)
at com.intellij.codeInsight.daemon.impl.DaemonCodeAnalyzerImpl$8.run(DaemonCodeAnalyzerImpl.java:3)
at com.intellij.util.Alarm$1.run(Alarm.java:98)
at com.intellij.util.Alarm$MyThread$1.run(Alarm.java:241)
at com.intellij.openapi.application.impl.LaterInvocatorEx$FlushQueue.run(LaterInvocatorEx.java:22)
at java.awt.event.InvocationEvent.dispatch(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:56)
at com.intellij.ide.IdeEventQueue.a(IdeEventQueue.java:91)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:132)
at java.awt.EventDispatchThread.pumpOneEventForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
ERROR - #com.intellij.util.Alarm -
ERROR - #com.intellij.util.Alarm - IntelliJ IDEA 5.1.2 Build #4267
ERROR - #com.intellij.util.Alarm - JDK: 1.5.0_07
ERROR - #com.intellij.util.Alarm - VM: Java HotSpot(TM) Client VM
ERROR - #com.intellij.util.Alarm - Vendor: Sun Microsystems Inc.
ERROR - #com.intellij.util.Alarm - OS: Windows XP
ERROR - #com.intellij.util.Alarm - Last Action:
ERROR - #com.intellij.util.Alarm -

3 comments

Hello Brian,

BS> Is there an example available that shows how to embed one language
BS> into another. I have tried embedding both Java and Javascript into
BS> my language and it failed with different errors each time.

I strongly recommend you to upgrade to IntelliJ IDEA 6.0. Lots of things
changed internally in this area, and I'm afraid we will not be able to provide
good support for problems that happen in version 5.1.2.

In particular, the new language injection API in version 6.0 may be a much
simpler solution for your problem (depending on what exactly your requirements
are).

--
Dmitry Jemerov
Software Developer
http://www.jetbrains.com/
"Develop with Pleasure!"


0

I found out that my program worked a lot better in 6.0. In particular, the Javascript fragments were parsed and Javascript errors are reported. But, I noticed two problems:

  • The Javascript fragments did not have syntax highlighting. How do I enable syntax highlighting for the embedded language?

  • The Javascript parser starts over in every fragment. For example, if I have:


-


function foo() {


host language comment

}

-



Then the Javascript parser treats the two fragments as totally seperate. I think that this is what the FileViewProvider interface is for. And, I that there is even a support class CompositeFileViewProvider and another one SingleRootFileViewProvider. It seems like I should extend CompositeFileViewProvider but it is not clear what I need to override for proper functioning. An example would be extremely helpful.

Also, will the FileViewProvider allow one to split an embedded token? For example:

-


/

  • Here is the beginning of a Javascript comment


Here is a host language comment

Here is the end of the initial Javascript comment */

-



Regards,
Brian

0

By the way, I am trying to use this for two projects that will be open-source and hopefully reusable by others. If you would like, we can work together to turn one or both of them into an example for the documentation.

  • Literate programming support: My idea is to create a "LiterateLanguage" that can then have any language embedded inside of it. Then, seperately one can implement "HaskellLanguage," "SchemeLanguage," seperately, and the support for the literate syntax style will be provided automatically.

  • Generic CPP (C Preprocessor) support. My idea here is to have a "CPPLanguage" that handles C preprocessor directives (#ifdef, #define, #include, etc.) Blocks of code between directives would be handled by the embedded language. The idea here is that the same CPP framework could be used by plugins for any language that uses CPP (C, C++, Haskell, etc.)

0

Please sign in to leave a comment.