Trying to make EOL significant in my grammar.. Follow
Answered
Here is my grammar:
{
tokens=[
eol="regexp:[\n\r]+"
eof='<<EOF>>'
literal="regexp:[^,\n'\"][^,\n]*"
string="regexp:('([^'\\]|\\.)*'|\"([^\"\\]|\\.)*\")"
comma=','
]
}
root ::= line *
line ::= !<<eof>> value (comma value)* ( eol | <<eof>>)
value ::= (string | literal)?
Here is my autogenerated .flex file (at this early stage, I prefer to keep it autogenerated): https://github.com/OptimisticLock/CsvIntelliJPlugin/blob/master/src/com/dreambox/csv/_CsvLexer.flex
Here is the error I am getting (I need this to be valid input).
The rest of the plugin can be found at the above Github link; though it does not contain much else at this point. FWIW, I did disable whitespaces in Parser Definition:
@NotNull
public TokenSet getWhitespaceTokens() {
return TokenSet.EMPTY;
}
Not sure how to suppress the following lines in the autogenerated .flex:
EOL=\R
WHITE_SPACE=\s
Anyway, it's a moot point as editing out those lines does not fix the problem.
Thank you!
Ирина
Please sign in to leave a comment.
Actually, please ignore the error above. Somehow in the latest version (on github) I got rid of the compilation error, but now all my csv file is one big huge PsiElement:
This seems correct:
This seems correct, too:
And this:
But this is a problem:
So looks like my AST tree is correct, but somehow it does not transfer to a correct Psi tree?
Now I recall that I've been going back and forth between compilation error and a Psi Tree with all null children for a while last time I touched this.
Maybe this will help:
Which sort of makes sense in light of this:
Assertion failed: Element(LINE)
java.lang.Throwable
at com.intellij.openapi.diagnostic.Logger.assertTrue(Logger.java:168)
at com.intellij.psi.impl.source.tree.CompositeElement.getChildrenAsPsiElements(CompositeElement.java:473)
at com.intellij.psi.impl.source.PsiFileImpl.getChildren(PsiFileImpl.java:831)
at com.dreambox.ext.MyTreeElement.getChildren(MyStructureViewBuilderFactory.java:88)
at com.intellij.ide.util.treeView.smartTree.TreeElementWrapper.initChildren(TreeElementWrapper.java:49)
at com.intellij.ide.util.treeView.smartTree.CachingChildrenTreeNode.b(CachingChildrenTreeNode.java:203)
at com.intellij.ide.util.treeView.smartTree.CachingChildrenTreeNode.a(CachingChildrenTreeNode.java:51)
at com.intellij.ide.util.treeView.smartTree.CachingChildrenTreeNode.getChildren(CachingChildrenTreeNode.java:44)
at com.intellij.ide.structureView.newStructureView.StructureViewComponent$StructureViewTreeElementWrapper.getChildren(StructureViewComponent.java:826)
at com.intellij.ide.util.treeView.smartTree.SmartTreeStructure.getChildElements(SmartTreeStructure.java:62)
at com.intellij.ide.util.treeView.AbstractTreeUi$31.perform(AbstractTreeUi.java:1617)
at com.intellij.ide.util.treeView.TreeRunnable.run(TreeRunnable.java:36)
at com.intellij.ide.util.treeView.AbstractTreeUi.execute(AbstractTreeUi.java:1835)
at com.intellij.ide.util.treeView.AbstractTreeUi.getChildrenFor(AbstractTreeUi.java:1614)
at com.intellij.ide.util.treeView.AbstractTreeUi.access$4500(AbstractTreeUi.java:64)
at com.intellij.ide.util.treeView.AbstractTreeUi$43.perform(AbstractTreeUi.java:2699)
at com.intellij.ide.util.treeView.TreeRunnable.run(TreeRunnable.java:36)
at com.intellij.ide.util.treeView.AbstractTreeUi.execute(AbstractTreeUi.java:1835)
at com.intellij.ide.util.treeView.AbstractTreeUi.access$2500(AbstractTreeUi.java:64)
at com.intellij.ide.util.treeView.AbstractTreeUi$54$1.perform(AbstractTreeUi.java:3376)
at com.intellij.ide.util.treeView.TreeRunnable.run(TreeRunnable.java:36)
at com.intellij.ide.util.treeView.AbstractTreeBuilder$2.perform(AbstractTreeBuilder.java:459)
at com.intellij.ide.util.treeView.TreeRunnable.run(TreeRunnable.java:36)
at com.intellij.openapi.application.impl.ApplicationImpl.runReadAction(ApplicationImpl.java:877)
at com.intellij.ide.util.treeView.AbstractTreeBuilder.runBackgroundLoading(AbstractTreeBuilder.java:456)
at com.intellij.ide.util.treeView.AbstractTreeUi$54.perform(AbstractTreeUi.java:3367)
at com.intellij.ide.util.treeView.TreeRunnable.run(TreeRunnable.java:36)
at com.intellij.openapi.progress.impl.CoreProgressManager$3.run(CoreProgressManager.java:179)
at com.intellij.openapi.progress.impl.CoreProgressManager.a(CoreProgressManager.java:568)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:519)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:54)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:164)
at com.intellij.ide.util.treeView.AbstractTreeUi$55.perform(AbstractTreeUi.java:3429)
at com.intellij.ide.util.treeView.TreeRunnable.run(TreeRunnable.java:36)
at com.intellij.ide.util.treeView.AbstractTreeUi.lambda$new$0(AbstractTreeUi.java:98)
at com.intellij.util.concurrency.QueueProcessor.lambda$null$0(QueueProcessor.java:106)
at com.intellij.util.concurrency.QueueProcessor.runSafely(QueueProcessor.java:223)
at com.intellij.util.concurrency.QueueProcessor.lambda$wrappingProcessor$1(QueueProcessor.java:106)
at com.intellij.util.concurrency.QueueProcessor.lambda$null$2(QueueProcessor.java:203)
at com.intellij.util.concurrency.QueueProcessor.runSafely(QueueProcessor.java:223)
at com.intellij.util.concurrency.QueueProcessor.lambda$startProcessing$3(QueueProcessor.java:203)
at com.intellij.openapi.application.impl.ApplicationImpl$2.run(ApplicationImpl.java:309)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
OK, problem solved by adding this line to bnf:
I think it calls for changing both Grammar-Kit and IntelliJ core to either show an error or recover gracefully when this line is omitted.
And now I was able to duplicate the same compilation error again, if you'd like to take a look at it. It's not urgent for me since I've now got a working version, but perhaps it's a grammar-kit bug? It's in the branch "error" of the same repo.
You have WHITE_SPACE rule before the CSVEOL one in _CsvLexer.flex, that explains why \n is treated as a whitespace and not as you expect. Please try reordering those rules.
BTW line separators are normalized to \n in the PSI, so you don't need to handle \r.