PsiParser - complex element

how should custom PsiParser implementation parse complex elements?

let's say, bnf looks like this:

xFile ::= line*

line ::= KEY


there is
XLineImpl extends ASTWrapperPsiElement

yet there is no wrapper for KEY element

auto generated XTypes.Factory  can createElement for XLine but not for KEY

so if parser does PsiBuilder.mark() twice (for LINE and KEY), createElement throws an error when trying to create KEY

if parser only mark() once and possibly remapCurrentToken, we only see either LINE or KEY in the resulting tree.


how do we parse a valid line to
    LINE
        KEY

?    


update:
if auto generated XTypes.Factory can createElement for both XLine and for KEY like this:

return new ASTWrapperPsiElement(node);


we get past create element error

however adding LINE marker around KEY
     with Marker.precede(), Marker.done()
does not add LINE node to the tree

15 comments
Comment actions Permalink
Alternatively, element type can implement ICompositeElementType and return proper composite node in createCompositeNode
0
Comment actions Permalink

Thank you Maxim!

well it is auto generated factory which throws an error.

createElement is "patched" now. But I am still unable to wrap KEY in LINE by calling precede() and done() on PsiBuilder.


1) when looking at this diagram, it appears that calling mark() and done() twice would create a complex element:
http://www.jetbrains.org/intellij/sdk/docs/reference_guide/custom_language_support/implementing_parser_and_psi.html


2) this other example:
https://upsource.jetbrains.com/idea-community/file/1731d054af4ca27aa827c03929e27eeb0e6a8366/plugins%2Fproperties%2Fproperties-psi-impl%2Fsrc%2Fcom%2Fintellij%2Flang%2Fproperties%2Fparsing%2FPropertiesParser.java

too calls builder.mark() twice

  final PsiBuilder.Marker rootMarker = builder.mark();
    final PsiBuilder.Marker propertiesList = builder.mark();
    while (!builder.eof()) {
      Parsing.parseProperty(builder);
    }
    propertiesList.done(PropertiesElementTypes.PROPERTIES_LIST);
    rootMarker.done(root);



I must be missing something.

0
Comment actions Permalink
IElementType LINE = new XElementType("LINE");
 
ICompositeElementType LINE = new XElementType("LINE");
0
Comment actions Permalink

I think your should be more clear in your problem / goal description.
If are going to stay on bnf level then lets look for existing project (e.g. https://github.com/JetBrains/intellij-plugins/tree/master/Dart) or ask GrammarKit specific question in https://github.com/JetBrains/Grammar-Kit/issues. Btw, it is not wise to fix the parser / types in such case.
Another (but more complex) option is go hand written parser way and my previous comment on your api question was based on assumption that you are going this way.

0
Comment actions Permalink

each Marker.mark / done will produce distinct composite ast node with type as given in argument.
There are two done calls in properties parser as AST structure should have two composite elements: inner, with

type PropertiesElementTypes.PROPERTIES_LIST and outer, with type PropertiesElementTypes.FILE
0
Comment actions Permalink

you are right. Let's narrow this down.

I am trying to implement PsiParser from 0.

However I am using GrammarKit generated files as a guide. I assume that some GrammarKit autogenerated files (not the parser of course) may still be reused with this new custom parser, no?

I may need to write my own XLineImpl, XKeyImpl etc but I am walking one step at a time (walk before I run ;))


my problem at this stage is:

is this possible to create composite element e.g.
LINE
     KEY

using nothing but PsiBuilder

?

0
Comment actions Permalink

> each Marker.mark / done will produce distinct composite ast node with type as given in argument.

This is what I expect, too. This does not happen. ?:|

Outer marker has no effect.

IElementType type passed to Marker.done(type) is simply ignored.

0
Comment actions Permalink

Maxim, my apologies.

It turns out that PsiBuilder produces a correct (as expected) ASTNode.

However by the time the tree is displayed in PsiViewer, it appears differently from PsiBuilder output.

The problem lies elsewhere.


Thank you very much for your time, Maxim! I appreciate this.

0
Comment actions Permalink

It is ok to reuse generated code until you decide to generate it again :)
Yes, custom node implementation is possible, as I described per fixing factory or via implementing

ICompositeElementType
0
Comment actions Permalink

PsiViewer shows PSI, not ast nodes

0
Comment actions Permalink

> It is ok to reuse generated code until you decide to generate it again :)

well autogen code was tweaked to test parser.

I will implement ICompositeElementType. Actually, will write custom factory for

 
public PsiElement createElement(ASTNode node)


Thank you Maxim.

0
Comment actions Permalink

> PsiViewer shows PSI, not ast nodes

a. However GrammarKit somehow succeeds to keep Psi tree very similar to AST tree. At least, every leaf (e.g. KEY) from bnf structure appears in PsiViewer.

LINE
     KEY

- just what I am trying to achieve.

0
Comment actions Permalink

AstWrapperPsiElement skips noncomposites when PsiElement.getChildren() is called. However custom code visiting all PSI tree structure will see tokens. So depending on how viewer is implemented it will see slightly different structure.
As reference I'd always use com.intellij.psi.impl.DebugUtil.treeToString - we use it for parsing tests

0
Comment actions Permalink

com.intellij.psi.impl.DebugUtil.treeToString

cheers! this certainly helps.

0
Comment actions Permalink

found the source of my woes.

I omitted these two lines:

     ...
  final PsiBuilder.Marker rootMarker = builder.mark();
     ...
  rootMarker.done(root);
     ...



from this example:

https://upsource.jetbrains.com/idea-community/file/1731d054af4ca27aa827c03929e27eeb0e6a8366/plugins%2Fproperties%2Fproperties-psi-impl%2Fsrc%2Fcom%2Fintellij%2Flang%2Fproperties%2Fparsing%2FPropertiesParser.java

this, and the GrammarKit autogen'ed XTypes.Factory.createElement needs to be decorated to





return new ASTWrapperPsiElement(node);


instead of

throw new AssertionError("Unknown element type: " + type);



sorry about the confusion. :8}

0

Please sign in to leave a comment.