Unexpected error with Syntax Highlighter

Looks like your lexer is not implemented correctly. Please make sure that the tokens returned by your lexer cover the entire length of the file and do not intersect with each other.

Well, it could be but I then do not understand while the same lexer is working when Syntax highlighting is disabled: it is the same lexer specified in the Parser Definition and I can see using PSI View that the nodes are parsed correctly.

My lexer is an ANTLR Lexer wrapped using code from the intellij-antlr-plugin so it seems weird to me that is causing this issue.

Your lexer is skipping regions of text for which it does not return a type token. This is the error that is being reported.

For syntax highlighting your lexer has to return tokens for every character in the text. You cannot skip any areas. For those areas that no token is needed it can return a WHITESPACE token.

However, if the highlighter encounters a skipped region you get the assertion error that you got.

I had that error in my hand rolled lexer for the MultiMarkdown plugin. So this is based on experience.

Thanks Vladimir, you are right!
I solved by using two different lexer (one for AST building, the other one for Syntax Highlight).

I would say that documentation and error message could be improved in this case.

Note that the requirement for a lexer to return a sequence of tokens covering the entire range of the file is exactly identical for the syntax highlighting lexer and the parsing lexer. Even though you might not see any assertions right away, you will see broken functionality and exceptions later on if your PSI contains holes.

The documentation is there but it is hard to absorb it in its entirety. I think that it is also more of an issue of being able to locate the relevant sources.

After I figured out what the issue was with my lexer I did reread the documentation: http://www.jetbrains.org/intellij/sdk/docs/reference_guide/custom_language_support/implementing_lexer.html and it clearly states this fact:

Thank you both: great answers!

Personally I think  it would be great to be able to reuse parsers written with other technologies (Antlr, JavaCC, etc.) but it seems hard because of the special requirements (lexers which emit tokens useless for parsers, the possibility to arbitrary roll back a lexer to a certain state).
I think it makes maintaining a plugin for IntelliJ a big effort.

I have previous experience developing pluging for Eclipse but there the community was larger so it was easier to find help and examples. I would really love to find more people working on IntelliJ plugins and share their knowledge.

I don't know how Eclipse handles PSI type refactoring but these requirements make sense when you see it from the perspective that manipulation of the PSI tree translates into manipulation of the underlying source. This is an awesome feature that I am working on taking advantage of, in the near future.

IntelliJ has AMAZING features with easy implementation on the plugin side but the learning curve is a challenge. There are a lot of places where relevant information can be found and that makes it a needle in the haystack scenario. This is especially true of finding the right source files in the community edition. It is all available but just like the internet, it requires a search engine to organize the search results by relevance and related information.

I agree with you that it would be nice to have more knowledge sharing between plugin developers and with 1000+ plugins there should be enough of us to make it work. The challenge is not adding more information but in finding what is already well documented and consolidating it. I also think that using existing online community tools for this would be a step in the right direction.

For example, I would include more details in getting the intellij-community project set-up so it builds. It took me 4 attempts before I got it right. Not difficult in the end but the documentation assumed that the reader builds Java projects for breakfast and a terse mention of the right terminology is enough. I have not worked with Java since 2003. It took a while to figure out exactly what was meant in the terse description through trial and error. This is where community input would be of great benefit. I would be willing to add to the documentation. However, the pages in question have no ability to do so right in the page and they don't show previously contributed community content. Instead, I am asked to go this discussion group to make a suggestion or ask a question. This raises the 'barrier of entry' of participatiion to unacceptable level because:

1. I have to create references back to the page in question. Which means I have to go between pages to get the job done right. Minor, but annoying.
2. I don't know what has already been asked or suggested. Which means I have to search this discussion and filter out 99.9% because it has nothing to do with the documentation page that I am concerned with. Now that is a show stopper. I simply don't have the time to do it right, so I won't do it at all.
3. I am in the middle of something that needs to be addressed, I don't have time for more distraction than absolutely necessary. A few times I actually considered doing it but gave up because it would make me loose track of what I was doing when a simple in page comment post would do.

Immediacy of feedback from the community is absolutely required for maximum community participation. PHP online documentation quite often has more relevance in the comments than the official text. Ability to add comments and suggestions right in the relevant is essential for the following reasons:

1. It makes in very convenient to participate right there and then with minimal effort.
2. It makes the previous questions and clarifications made by the community available for others to see. This eliminates duplicates and allows others to answer with minimal effort.
3. It makes it easier to incorporate relevant suggestions on the JetBrains side, and eliminate any pressure in doing it ASAP. The docs may be out date but the comments and suggestions compensate for it.

Maybe the plugin development documentation should be moved to GitHub Wiki for the intellij-community project. It would add all these community features for free and would become the goto source for plugin developers. Enticing greater community participation. A win-win for everyone.

Another example of improving ease of participation is StackOverflow. It has a tag for 'intellij-plugin' but it only has 223 questions total. It does not appear to be the goto source for plugin dev help. I like the StackOverflow UI, features, search capability and especially the site response speed and google search results but I doubt the community will make the switch without official IntelliJ adoption of that platform over this discussion. The experts from this discussion are essential in that forum so that developers can start using it as the goto place to get their answers. I suspect that with time the experts' participation would be reduced to questions that merit their level of expertise, which I think they would consider a welcome change. Using existing community tools like StackOverflow is desirable for the same reason JetBrains uses GitHub.

To summarize this long winded post, I think it would benefit IntelliJ platform if an effort was made to consolidate the plugin development community. By their own admission, plugin development learning curve and adoption leaves better to be desired. I think the solution is not more documentation but in consolidation of what is already well documented and enticing the community of existing developers to participate by making it easier through accepted and familiar platforms.

Thanks for sharing your point of view.

I agree that Jetbrains's products are great and I have interacted a bit with them when working on Mbeddr (a project based on MPS): I found them quite nice and responsive

IDE Plugin development tends to be hard on all the platforms but it really struck me the lack of a community for IntelliJ: it is becoming more and more the IDE of reference and more plugins are being developed, still I find quite hard to find blogs, tutorials and other source of informations outside the jetbrains website.

In my particular I think that it could be extremely useful and not much complicate to provide adapters for lexers/parsers generated by ANTLR or JavaCC. It would make plugin development much easier.

In addition to that I think that Xtext is an amazing example of what it can be done to simplify editor development: by simply specifying the grammar you get for free syntax highlighint and syntactical auto-completion. Doing that on IDEA seems much more complicate to me. Sure you may want to personalize these aspects but I think that sensible defaults would really help people get started and lower the barriers.

For the rest... I think pugin developers should blog more :)

Have you looked into grammar-kit? https://github.com/JetBrains/grammar-kit

Yes, I am familiar with Grammar-Kit but 1) it seems a solution specific to IntelliJ while using things like ANTLR I could use the same parser also in the compiler of my language 2) it does not seem to automate much: I really like the approach of having sensible defaults as in XText. For example it is easy to find out keywords or provide auto-completion based on syntax. As far as I understood in IntelliJ I have to specify everything manually: while it is great to have control I think we could get more for free from the bnf.

I have this issue of Unexpected exception during symbol building (rebuildModuleMaps) on starting up my Andriod studio 2.3.3 and is also displaying a null pointer exception, I really need support to help me resolve this issue.

Unexpected exception during symbol building (rebuildModuleMaps)
java.lang.NullPointerException
at com.jetbrains.cidr.lang.workspace.OCWorkspaceManager.getWorkspace(OCWorkspaceManager.java:12)
at com.jetbrains.cidr.modulemap.symbols.ModuleMapCacheBuilder.getHeaderRoots(ModuleMapCacheBuilder.kt:39)
at com.jetbrains.cidr.modulemap.symbols.ModuleMapCacheBuilder.getTasks(ModuleMapCacheBuilder.kt:19)
at com.jetbrains.cidr.lang.symbols.symtable.OCSymbolTablesBuildingActivity.lambda$buildModuleMapsInternal$5(OCSymbolTablesBuildingActivity.java:139)
at com.intellij.openapi.application.impl.ApplicationImpl.runReadAction(ApplicationImpl.java:848)
at com.jetbrains.cidr.lang.symbols.symtable.OCSymbolTablesBuildingActivity.buildModuleMapsInternal(OCSymbolTablesBuildingActivity.java:138)
at com.jetbrains.cidr.lang.symbols.symtable.OCSymbolTablesBuildingActivity.lambda$rebuildModuleMaps$4(OCSymbolTablesBuildingActivity.java:129)
at com.jetbrains.cidr.lang.symbols.symtable.OCSymbolTablesBuildingActivity$1.performInDumbMode(OCSymbolTablesBuildingActivity.java:180)
at com.intellij.openapi.project.DumbServiceImpl.lambda$runSingleTask$10(DumbServiceImpl.java:474)
at com.intellij.openapi.progress.impl.CoreProgressManager$3.run(CoreProgressManager.java:170)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:494)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:443)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:54)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:155)
at com.intellij.openapi.project.DumbServiceImpl.runSingleTask(DumbServiceImpl.java:467)
at com.intellij.openapi.project.DumbServiceImpl.access$300(DumbServiceImpl.java:52)
at com.intellij.openapi.project.DumbServiceImpl$4.run(DumbServiceImpl.java:439)
at com.intellij.openapi.progress.impl.CoreProgressManager$TaskRunnable.run(CoreProgressManager.java:635)
at com.intellij.openapi.progress.impl.CoreProgressManager$3.run(CoreProgressManager.java:170)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:494)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:443)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:54)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:155)
at com.intellij.openapi.progress.impl.ProgressManagerImpl$1.run(ProgressManagerImpl.java:128)
at com.intellij.openapi.application.impl.ApplicationImpl$2.run(ApplicationImpl.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

java.lang.NullPointerException
at com.jetbrains.cidr.lang.workspace.OCWorkspaceManager.getWorkspace(OCWorkspaceManager.java:12)
at com.jetbrains.cidr.lang.symbols.symtable.OCSymbolTablesBuildingActivity.clearAllSymbolDependentCaches(OCSymbolTablesBuildingActivity.java:395)
at com.jetbrains.cidr.lang.symbols.symtable.OCSymbolTablesBuildingActivity.access$100(OCSymbolTablesBuildingActivity.java:49)
at com.jetbrains.cidr.lang.symbols.symtable.OCSymbolTablesBuildingActivity$3.run(OCSymbolTablesBuildingActivity.java:255)
at com.jetbrains.cidr.lang.symbols.symtable.OCSymbolTablesBuildingActivity$10$1.run(OCSymbolTablesBuildingActivity.java:434)
at com.intellij.openapi.application.impl.ApplicationImpl.runWriteAction(ApplicationImpl.java:898)
at com.jetbrains.cidr.lang.symbols.symtable.OCSymbolTablesBuildingActivity$10.run(OCSymbolTablesBuildingActivity.java:430)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:726)
at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:795)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:631)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:387)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)