Transient Lexer Errors In My Custom Language Plugin When Starting The IDE

Hello Jon,

As a general principle, a custom languge lexer has to be able to index any
text to the end, and produce bad character tokens for all characters that
weren't valid. I don't know why you experience the problem specifically when
indexing (this might be a bug), but the lexer needs to be fixed in any case.
:)

This seems to happen only on startup of the IDE. I will occasionally
get errors from the lexer of my custom language plugin. The errors
don't seem consistant, and if I open the file my plugin has no problem
lexing and parsing it. It is only at startup indexing it seems.

Does anyone have any insight into what could be wrong or suggestions
on how to avoid the errors?

Here's an example of one:

2011-01-15 00:17:01,503    INFO -
indexing.UnindexedFilesUpdater - Unindexed files update started
2011-01-15 00:17:02,500   ERROR -
api.project.CacheUpdateSession - Error while indexing C:\World of
Warcraft\Blizzard Interface Data (enUS)\FrameXML\LFGFrame.lua
To reindex this file IDEA has to be restarted
java.lang.Error: com.sylvanaar.idea.Lua.lang.lexer._LuaLexer: Error:
could not match input
at
com.sylvanaar.idea.Lua.lang.lexer._LuaLexer.zzScanError(_LuaLexer.java
:476)
at
com.sylvanaar.idea.Lua.lang.lexer._LuaLexer.advance(_LuaLexer.java:928
)
at com.intellij.lexer.FlexAdapter.locateToken(FlexAdapter.java:85)
at
com.intellij.lexer.FlexAdapter.getTokenType(FlexAdapter.java:54)
at
com.intellij.lang.cacheBuilder.DefaultWordsScanner.processWords(Defaul
tWordsScanner.java:56)
at
com.intellij.psi.impl.cache.impl.id.IdTableBuilding$WordsScannerFileTy
peIdIndexerAdapter.map(IdTableBuilding.java:248)
at
com.intellij.psi.impl.cache.impl.id.IdTableBuilding$WordsScannerFileTy
peIdIndexerAdapter.map(IdTableBuilding.java:236)
at
com.intellij.psi.impl.cache.impl.id.IdIndex$4.map(IdIndex.java:79)
at
com.intellij.psi.impl.cache.impl.id.IdIndex$4.map(IdIndex.java:74)
at
com.intellij.util.indexing.MapReduceIndex.update(MapReduceIndex.java:2
00)
at
com.intellij.util.indexing.FileBasedIndex$21.run(FileBasedIndex.java:1
377)
at
com.intellij.openapi.progress.impl.ProgressManagerImpl.executeNonCance
lableSection(ProgressManagerImpl.java:142)
at
com.intellij.util.indexing.FileBasedIndex.a(FileBasedIndex.java:1374)
at
com.intellij.util.indexing.FileBasedIndex.indexFileContent(FileBasedIn
dex.java:1340)
at
com.intellij.util.indexing.UnindexedFilesUpdater.processFile(Unindexed
FilesUpdater.java:56)
at
com.intellij.openapi.project.CacheUpdateSession.processFile(CacheUpdat
eSession.java:102)
at
com.intellij.openapi.project.CacheUpdateRunner$MyRunnable$1.run(CacheU
pdateRunner.java:223)
at
com.intellij.openapi.application.impl.ApplicationImpl.runReadAction(Ap
plicationImpl.java:760)
at
com.intellij.openapi.project.CacheUpdateRunner$MyRunnable.run(CacheUpd
ateRunner.java:227)
at
com.intellij.openapi.application.impl.ApplicationImpl$6.run(Applicatio
nImpl.java:382)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441
)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
tor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:908)
at java.lang.Thread.run(Thread.java:619)
at
com.intellij.openapi.application.impl.ApplicationImpl$1$1.run(Applicat
ionImpl.java:130)
Here is an example of another:

Error while indexing /Users/callahan/Projects/AckisRecipeList/core.lua

To reindex this file IDEA has to be restarted:
com.sylvanaar.idea.Lua.lang.lexer._LuaLexer: Error: could not match
input

java.lang.Error: com.sylvanaar.idea.Lua.lang.lexer._LuaLexer: Error:
could not match input

     at
com.sylvanaar.idea.Lua.lang.lexer._LuaLexer.zzScanError(_LuaLexer.java
:476)

     at
com.sylvanaar.idea.Lua.lang.lexer._LuaLexer.advance(_LuaLexer.java:928
)

     at
com.intellij.lexer.FlexAdapter.locateToken(FlexAdapter.java:85)

     at
com.intellij.lexer.FlexAdapter.getTokenType(FlexAdapter.java:54)

     at
com.intellij.lang.cacheBuilder.DefaultWordsScanner.processWords(Defaul
tWordsScanner.java:56)

     at
com.intellij.psi.impl.cache.impl.id.IdTableBuilding$WordsScannerFileTy
peIdIndexerAdapter.map(IdTableBuilding.java:248)

     at
com.intellij.psi.impl.cache.impl.id.IdTableBuilding$WordsScannerFileTy
peIdIndexerAdapter.map(IdTableBuilding.java:236)

     at
com.intellij.psi.impl.cache.impl.id.IdIndex$4.map(IdIndex.java:79)

     at
com.intellij.psi.impl.cache.impl.id.IdIndex$4.map(IdIndex.java:74)

     at
com.intellij.util.indexing.MapReduceIndex.update(MapReduceIndex.java:2
00)

     at
com.intellij.util.indexing.FileBasedIndex$21.run(FileBasedIndex.java:1
377)

     at
com.intellij.openapi.progress.impl.ProgressManagerImpl.executeNonCance
lableSection(ProgressManagerImpl.java:142)

     at
com.intellij.util.indexing.FileBasedIndex.updateSingleIndex(FileBasedI
ndex.java:1374)

     at
com.intellij.util.indexing.FileBasedIndex.indexFileContent(FileBasedIn
dex.java:1340)

     at
com.intellij.util.indexing.UnindexedFilesUpdater.processFile(Unindexed
FilesUpdater.java:56)

     at
com.intellij.openapi.project.CacheUpdateSession.processFile(CacheUpdat
eSession.java:102)

     at
com.intellij.openapi.project.CacheUpdateRunner$MyRunnable$1.run(CacheU
pdateRunner.java:223)

     at
com.intellij.openapi.project.CacheUpdateRunner$MyRunnable.run(CacheUpd
ateRunner.java:230)

     at
com.intellij.openapi.project.CacheUpdateRunner$3.run(CacheUpdateRunner
.java:248)

     at
com.intellij.openapi.application.impl.ApplicationImpl$6.run(Applicatio
nImpl.java:382)

     at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441
)

     at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

     at
java.util.concurrent.FutureTask.run(FutureTask.java:138)

     at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
tor.java:886)

     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:908)

     at java.lang.Thread.run(Thread.java:680)

     at
com.intellij.openapi.application.impl.ApplicationImpl$1$1.run(Applicat
ionImpl.java:130)

---
Original message URL:
http://devnet.jetbrains.net/message/5283693#5283693

--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"

I had exactly the same issue with my CoffeeScript plugin. I resolved it with the following method:

1. Generate the lexer
2. Change the exception message within the generated lexer to include the current state and text (zzScanError method)
3. Start the ide
4. Open the log file and see in what state the plugin failed and what the text was
5. Change the lexer to catch that case and start over from 1. until all exceptions are gone

In my case there were newlines in states where they shouldn't apear.

Michael

Thanks for the tip. It sounds like a great way to debug the issue. Also good to know I am not the only one with this issue!

Have had the same issues with my plugin, and learned that startup indexing is somehow using the lexers asynchronously, or the lifecycle of a lexing event is not defined properly (as they seem to be overlapping). So if you share the same instance of the flex lexer, this may work fine during sequential or single file lexing, but not when multiple threads are using it (i.e in case of the indexing procedure). To make the long story short, make sure you are always creating a new lexer whenever lexing is needed, rather than reusing the same instance.