Parser w/o ASTNode

To get started, I would like to be able to have some functionality from my parser. However, since it doesn't currently spit out AST trees, I would like to be able to just hightlight any errors. So my question is: Is it possible to get a handle to the error handler and hightlight code if I can find out the line/col number, etc? If so, could I get a pointer in the right direction? Or do I haveto return the ASTNode and let it be handled for me. It seems as though implementing AST trees would effectively be re-implementing the parser.

thanks again!
adam

17 comments
Comment actions Permalink

Yep, that's possible.

You'll have to implement getAnnotator() method in your language class and
return Annotator interface implementation. Even if you do not implement the
full blown parser you still can have single PsiElement for the whole file
that will be passed to Annotator.annotate() method along with AnnotationHolder
method. Latter can be populated with error or warning messages using create<*>Annotation()
methods.

-


Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

To get started, I would like to be able to have some functionality
from my parser. However, since it doesn't currently spit out AST
trees, I would like to be able to just hightlight any errors. So my
question is: Is it possible to get a handle to the error handler and
hightlight code if I can find out the line/col number, etc? If so,
could I get a pointer in the right direction? Or do I haveto
return the ASTNode and let it be handled for me. It seems as though
implementing AST trees would effectively be re-implementing the
parser.

thanks again!
adam




0
Comment actions Permalink

Thanks Maxim! This helps a lot. I'm still not sure what to do in the parse method. Returning null doesn't appear to be ok;)

adam

0
Comment actions Permalink

I've found reasonable "mock" implementation in Groovy plugin by Dierk at
https://svn.canoo.com/trunk/groovyj/src/java/groovy/idesupport/idea/parsing/GroovyParser.java
Hopefully I'm not doing anything bad by posting this link as it have been
already posted here in the list.
-


Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

Thanks Maxim! This helps a lot. I'm still not sure what to do in the
parse method. Returning null doesn't appear to be ok;)

adam




0
Comment actions Permalink

Thanks a lot for all your help!

adam

0
Comment actions Permalink

Maxim,
I was able to more done using this method. However, I am getting many exceptions and out of mem errors when trying to load even an empty file. Could you possibly provide a little more info on the necessary methods in the lexer? I think I must be doing something wrong there. I am currently creating a new lexer each time the parse method gets called and passing in the text from the builder. Also, I am not doing anything in the 3 start methods because of this. If you can provide some info, I can relay this info to the wiki for others to use and it won't be lost.

0
Comment actions Permalink

Adam,

creating new Lexer every time you need one or required for is fine. Not implementing
start methods - that's what causes problems most probably. Since Lexer is
an interface there's no way to enforce initialization and pass required parameters
at object creation time. That's why initialization is performed in start
methods.


-


Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

Maxim,
I was able to more done using this method. However, I am getting
many exceptions and out of mem errors when trying to load even an
empty file. Could you possibly provide a little more info on the
necessary methods in the lexer? I think I must be doing something
wrong there. I am currently creating a new lexer each time the parse
method gets called and passing in the text from the builder. Also, I
am not doing anything in the 3 start methods because of this. If you
can provide some info, I can relay this info to the wiki for others to
use and it won't be lost.




0
Comment actions Permalink

ok - but what exactly needs to be done in the start methods? Should I wait and only create the new lexer here?

0
Comment actions Permalink

And sorry for barrage of questions. I'm getting close to a deadline...

0
Comment actions Permalink

/**

  • Prepare for lexing character data from <code>buffer</code> passed. Lexing

should be performed starting from offset 0 and

  • terminated (EOFed) at offset buffer.length. Internal lexer state is

supposed to be initial. YY_INITIAL for JLex and JFlex generated

  • lexer.

  • @param buffer character data for lexing.

*/
void start(char[] buffer);

/**

  • Prepare for lexing character data from <code>buffer</code> passed. Lexing

should be performed starting from offset startOffset>]]> and

  • terminated (EOFed) at offset <code>endOffset</code>. Internal lexer

state is supposed to be initial. YY_INITIAL for JLex and JFlex generated

  • lexer.

  • @param buffer character data for lexing.

  • @param startOffset offset to start lexing from

  • @param endOffset offset to stop lexing at

*/
void start(char[] buffer, int startOffset, int endOffset);

/**

  • Prepare for lexing character data from <code>buffer</code> passed. Lexing

should be performed starting from offset startOffset>]]> and

  • terminated (EOFed) at offset <code>endOffset</code>. Internal lexer

state is supposed to be initialState]]>. It is guaranteed

  • that value of initialState have been returned by {@link #getState()}

method of this Lexer at condition

startOffset=getTokenStart()
]]>

  • This method is used to incrementally relex changed characters using

lexing data acquired from this particular lexer sometime in the past.

  • @param buffer character data for lexing.

  • @param startOffset offset to start lexing from

  • @param endOffset offset to stop lexing at

*/
void start(char[] buffer, int startOffset, int endOffset, int initialState);

-


Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

ok - but what exactly needs to be done in the start methods? Should I
wait and only create the new lexer here?




0
Comment actions Permalink

That's OK. We're very interested for this API to be used. This way we'll
be able to make it better where it is still rough on edges.
What is the language you're implementing the plugin for if it's not a secret?

-


Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

And sorry for barrage of questions. I'm getting close to a
deadline...




0
Comment actions Permalink

And now it's clear Lexer interface has to be reworked. Thanks!
-


Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

That's OK. We're very interested for this API to be used. This way
we'll
be able to make it better where it is still rough on edges.
What is the language you're implementing the plugin for if it's not a
secret?
-------------------
Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

>> And sorry for barrage of questions. I'm getting close to a
>> deadline...
>>



0
Comment actions Permalink

Its actually a toy language for a master's project I'm working on called mini-java, which is used to generate native assembly from java. I chose this project because I wanted to get get some practice in doing a plugin before doing an industrial strength one at work.

And I really, really appreciate all the help - especially on Saturday!

0
Comment actions Permalink

Maxim,
I think I'm doing the right thing in the start methods, but when I try to load a simple file, I still can't get past the out of mem error. I also get the following:

java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at com.intellij.util.io.ByteBufferRADataInput.setPosition(ByteBufferRADataInput.java:24)
at com.intellij.util.io.ByteBufferMap.(ByteBufferMap.java:33) at com.intellij.psi.impl.cache.impl.idCache.ImmutableIdCacheImpl.a(ImmutableIdCacheImpl.java:61) at com.intellij.psi.impl.cache.impl.idCache.ImmutableIdCacheImpl.getFilesWithTodoItems(ImmutableIdCacheImpl.java:29) at com.intellij.psi.impl.cache.impl.idCache.IdCacheImpl.getFilesWithTodoItems(IdCacheImpl.java:94) at com.intellij.psi.impl.cache.impl.CacheManagerImpl.getFilesWithTodoItems(CacheManagerImpl.java:36) at com.intellij.psi.impl.cache.impl.CompositeCacheManager.getFilesWithTodoItems(CompositeCacheManager.java:20) at com.intellij.psi.impl.search.PsiSearchHelperImpl.findFilesWithTodoItems(PsiSearchHelperImpl.java:242) at com.intellij.ide.todo.AllTodosTreeBuilder.rebuildCache(AllTodosTreeBuilder.java:13) at com.intellij.ide.todo.TodoTreeBuilder.init(TodoTreeBuilder.java:192) at com.intellij.ide.todo.TodoView$2.createTreeBuilder(TodoView.java:0) at com.intellij.ide.todo.TodoPanel.(TodoPanel.java:42) at com.intellij.ide.todo.TodoView$2.]]>(TodoView.java:1)
at com.intellij.ide.todo.TodoView$1.run(TodoView.java:4)
at com.intellij.ide.startup.impl.StartupManagerImpl.a(StartupManagerImpl.java:7)

0
Comment actions Permalink

I think I narrowed down the problem. The code seems to be bouncing back and forth between getTokenStart and getTokenEnd infinitely. Any ideas what could cause this?

0
Comment actions Permalink

More info: It looks like builder.eof is never true. Is there a token or something I need to return to indicate this?

adam

0
Comment actions Permalink

Return null as getTokenType() when you finish.

More info: It looks like builder.eof is never true. Is there a token
or something I need to return to indicate this?

adam



0
Comment actions Permalink

Thanks Maxim, but I am still getting a OOB exception that I mentioned in the next thread. Any ideas?

adam

0

Please sign in to leave a comment.