Parser Startup Errors

I'm attempting to implement template support in the Gosu plugin.  Here's what a gosu template looks like:

"blah blah
  ${10 + 10}
blah blah"

My first stab is to lex this into three element types: content, script, content, and then use language injection on the script to get gosu support.  I know this has shortcomings and I don't think it will be the long term approach we use, but it will let me learn the IJ parsing model a bit and might lead to other insights.

I'm embarrassed to admit that I am having trouble even getting that simple parser off the ground.  The lexer appears to work fine, but when I transform it into a PSI tree, I end up with the first node in the file's children not having a link to the next one.  Here is the relevant parts of my parser:

  public PsiParser createParser( Project project )
  {
    return new PsiParser()
    {
      @NotNull
      @Override
      public ASTNode parse( IElementType root, PsiBuilder builder )
      {
        builder.setDebugMode( true );
        PsiBuilder.Marker rootMark = builder.mark();
        while( !builder.eof() )
        {
          PsiBuilder.Marker mark = builder.mark();
          IElementType tokenType = builder.getTokenType();
          mark.done( tokenType );
          builder.advanceLexer();
        }
        rootMark.done( root );
        return builder.getTreeBuilt();
      }
    };
  }

  @NotNull
  public PsiElement createElement( ASTNode node )
  {
    if( node.getElementType() == GosuTemplateTokenTypes.TEMPLATE_CONTENT_TYPE )
    {
      return new GosuTemplateContentPsiElement( node );
    }
    else if ( node.getElementType() == GosuTemplateTokenTypes.TEMPLATE_SCRIPT_TYPE )
    {
      return new GosuTemplateScriptPsiElement( node );
    }
    else
    {
      throw new IllegalStateException( "Could not create element for " + node );
    }
  }

Anything obvious I'm doing wrong here?  

Thanks,
Carson
9 comments
Comment actions Permalink

Did you make new IElementTypes for your PSI elements? It looks like you are using the lexer token types.

Have you installed the PSIViewer Addon? It is a must for debugging this sort of thing.

When you say "not having a link to the next one" what do you mean specifically.

0
Comment actions Permalink

Jon,

I'm using the IElementTypes from the tokenizer when I construct my admittedly simple ASTNodes.  In this case, the lexer is emitting all the content as a single token (including whitespace.)  So

foo  foo
${bar}

is emitted as the tokens "foo foo\n" and "${bar}.

I'm then building up a simple 1:1 PSI tree for them.

What's happening is that additional CompositeElements are ending up under the File PSI object for some reason.  I'm trying to figure out why.

I'll try the PSIViewer Addon and see what that shows me.

Thanks for your help!

Cheers,
Carson

0
Comment actions Permalink

Hmmm.  Using .collapse() rather than .done() in my parse seems to have fixed the issue.

0
Comment actions Permalink

Heh.  One step forward, one back.  Using collapse() fixes my PSI Tree (although I don't understand why it would) but now I get PSILeafElements in my PSITree.  

Yuck.

0
Comment actions Permalink

Alrighty, I've got things cleaned up.  I now see what you mean about me reusing the tokenizer IElementTypes: I just figured that since there was a 1:1 mapping, it wouldn't matter.  It apparently does.  ;)

Another error I discovered was that I picked some bad base classes for my PSI Elements.  Changing them to extend ASTWrapperPsiElement fixed a lot of stuff.

So I'm in a much better spot than a few hours ago.  However, given a template like  this:

foo foo${bar}

I end up with four, rather than two, ASTNodes under my file: I end up with a composite node, followed by a leaf node, then another composite node, and another leaf node. I expect there to only be two.  My "parser" code is pretty simple:

  public PsiParser createParser( Project project )
  {
    return new PsiParser()
    {
      @NotNull
      @Override
      public ASTNode parse( IElementType root, PsiBuilder builder )
      {
        builder.setDebugMode( true );
        PsiBuilder.Marker rootMark = builder.mark();
        while( !builder.eof() )
        {
          PsiBuilder.Marker mark = builder.mark();
          IElementType tokenType = builder.getTokenType();
          if( tokenType == GosuTemplateTokenTypes.TEMPLATE_CONTENT_TYPE )
          {
            mark.done( GosuTemplatePsiTypes.TEMPLATE_CONTENT_TYPE );
          }
          else
          {
            mark.done( GosuTemplatePsiTypes.TEMPLATE_SCRIPT_TYPE );
          }
          builder.advanceLexer();
        }
        rootMark.done( root );
        return builder.getTreeBuilt();
      }
    };
  }


I'll keep poking around, but if something obvious jumps out at anyone, many thanks.

Cheers,
Carson
0
Comment actions Permalink

The leaves are the lexer tokens I believe


This is a great time to install PsiViewer

0
Comment actions Permalink

:)

Yeah, PsiViewer has been a lot of help.

The problem was that I was calling done() *before* I advanced the tokenizer to the next token, so the token PsiElements were being created as siblings rather than children of the elements I was creating.

Thanks for all your help Jon.

Cheers,
Carson

0
Comment actions Permalink

BTW are you guys aware of IDEA's bundled in PSI Viewer (Tools | View PSI Structure) ?

0
Comment actions Permalink

Yes. It is not nearly as useful.

0

Please sign in to leave a comment.