builder.getTreeBuilt() OOB Exception Follow

The wild guess is your lexer returns empty (getTokenStart() == getTokenEnd())
lexems at some point or advance() method doesn't advance. Can I take a look
at sources? I guess I could investigate the problem much faster that way.

-

Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

I am trying to user builder.getTreeBuilt() to get an ASTNode to return
an ASTNode to the parse method of PsiParser, but it is returning the
OOB exception below. Has anyone had this problem or know what could
cause it?

thanks,
adam
ERROR - mand.impl.CommandProcessorImpl - Last Action:
EditorEnter
ERROR - mand.impl.CommandProcessorImpl - Current Command:
Typing
ERROR - mand.impl.CommandProcessorImpl -
java.lang.ArrayIndexOutOfBoundsException: 1
at java.lang.String.hashCode(String.java:1074)
at gnu.trove.TObjectHash.computeHashCode(Unknown Source)
at gnu.trove.TObjectHash.index(Unknown Source)
at gnu.trove.TObjectIntHashMap.get(Unknown Source)
at
com.intellij.psi.impl.source.CharTableImpl.getId(CharTableImpl.java:32
)
at
com.intellij.psi.impl.source.tree.LeafElementImpl.<init>(LeafElementIm
pl.java:22)
at
com.intellij.psi.impl.source.tree.LeafPsiElement.<init>(LeafPsiElement
.java:14)
at com.intellij.lang.impl.PsiBuilderImpl.a(PsiBuilderImpl.java:20)
at
com.intellij.lang.impl.PsiBuilderImpl.getTreeBuilt(PsiBuilderImpl.java
:112)

java.lang.ArrayIndexOutOfBoundsException: 1
at java.lang.String.hashCode(String.java:1074)
at gnu.trove.TObjectHash.computeHashCode(Unknown Source)
at gnu.trove.TObjectHash.index(Unknown Source)
at gnu.trove.TObjectIntHashMap.get(Unknown Source)
at com.intellij.psi.impl.source.CharTableImpl.getId(CharTableImpl.java:32)
at com.intellij.psi.impl.source.tree.LeafElementImpl.<init>(LeafElementImpl.java:22)
at com.intellij.psi.impl.source.tree.LeafPsiElement.<init>(LeafPsiElement.java:14)
at com.intellij.lang.impl.PsiBuilderImpl.a(PsiBuilderImpl.java:20)
at com.intellij.lang.impl.PsiBuilderImpl.getTreeBuilt(PsiBuilderImpl.java:112)

How the hell can String.hashCode() ever throw such an exception?
There must be something really strange going on.

Sascha

Sure - where should I send them?

Just in case, here it is:

<![CDATA[

import com.intellij.lexer.Lexer;
import com.intellij.psi.tree.IElementType;
import jacc.Settings;
import compiler.Handler;
import compiler.SimpleHandler;
import compiler.JavaSource;
import java.io.*;
import java.util.HashMap;

import mjc.MjcLexer;

/**
 * Mjc Lexer Wrapper
 * @author Adam Cabler
 */
public class MjcIJLexer implements Lexer {
	private Handler handler = new SimpleHandler();
	private PrintWriter out = new PrintWriter(System.out, true);
	private Settings settings = new Settings();
	private HashMap types;
	private int type;
	private IElementType iType;
	private MjcLexer lexer;

	public MjcIJLexer(Handler handler, Reader reader) {
		lexer = new MjcLexer(handler, new JavaSource(handler, "MJC Source", reader));
		type = lexer.nextToken();
		types = new HashMap();
	}

	public void setLexer(MjcLexer lexer) {
		this.lexer = lexer;
	}

	public void setType(int type) {
		this.type = type;
	}

	private IElementType addType(int type) {
		iType = new IElementType(Integer.toString(type), MjcSupportLoader.MJCFILE.getLanguage());
		types.put(new Integer(type), iType);
		return (IElementType)(types.get(new Integer(type)));
	}

	private IElementType getType(int type) {
		if(types.containsKey(new Integer(type))) {
			return (IElementType)types.get(new Integer(type));
		} else {
			return addType(type);
		}
	}

	public MjcLexer getLexer() {
		return lexer;
	}

	public final void start(char[] buffer) {
		start(((char[]) buffer), 0, ((char[]) buffer).length);
	}

	public final void start(char[] buffer, int startOffset, int endOffset) {
		start(((char[]) buffer), startOffset, endOffset, 0);
	}

	public void start(char[] buffer, int startOffset, int endOffset, int initialState) {
		String temp = "";
		for (int i = startOffset; i < endOffset; i++) {
			temp += buffer+;

		}
		lexer = new MjcLexer(handler, new JavaSource(handler, "MJC Source", new StringReader(temp)));
		type  = lexer.nextToken();
	}

	public int getState() {
		return 0;
	}

	public int getLastState() {
		return 0;
	}

	public IElementType getTokenType() {
		if(type == 0) {
			return null;
		}
		return getType(type);
	}

	public int getTokenStart() {
		return lexer.getPos().getColumn();
	}

	public int getTokenEnd() {
		int posCol = lexer.getPos().getColumn();
		String lex = lexer.getLexeme();
		if (lex == null) {
			return posCol + 1;
		}
		return posCol + lex.length();
	}

	public void advance() {
		type = lexer.nextToken();
	}

	public char[] getBuffer() {
		String lex = lexer.getLexeme();
		if (lex == null) {
			return new char[]{0};
		}
		return lexer.getLexeme().toCharArray();
	}

	public int getBufferEnd() {
		String lex = lexer.getLexeme();
		if (lex == null) {
			return 0;
		}
		return lexer.getLexeme().length();
	}

	public int getSmartUpdateShift() {
		return 0;  //To change body of implemented methods use File | Settings | File Templates.
	}

	public Object clone() {
		try {
			return super.clone();
		} catch (CloneNotSupportedException e) {
			return null;
		}
	}
}
]]>

1. getBuffer() should return char[] originally passed to start().
2. getBufferEnd() should return originally passed endOffset (or buffer.length)
3. Concatenating characters from char[] is very uneffective. Use CharArrayReader
if you need just reader instead.
4. getTokenStart()/getTokenEnd() should return offsets in original buffer,
not column number.
5. Lazy creation of IElementType is bad idea. Lexer should return same instances
of IElementTypes when run against same text. At least store them somewhere
globally, not per Lexer's instance.

-

Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

Just in case, here it is:

<![CDATA[
> import com.intellij.lexer.Lexer;
> import com.intellij.psi.tree.IElementType;
> import jacc.Settings;
> import compiler.Handler;
> import compiler.SimpleHandler;
> import compiler.JavaSource;
> import java.io.*;
> import java.util.HashMap;
> import mjc.MjcLexer;
> 
> /**
> * Mjc Lexer Wrapper
> * @author Adam Cabler
> */
> public class MjcIJLexer implements Lexer {
> private Handler handler = new SimpleHandler();
> private PrintWriter out = new PrintWriter(System.out, true);
> private Settings settings = new Settings();
> private HashMap types;
> private int type;
> private IElementType iType;
> private MjcLexer lexer;
> public MjcIJLexer(Handler handler, Reader reader) {
> lexer = new MjcLexer(handler, new JavaSource(handler, "MJC Source",
> reader));
> type = lexer.nextToken();
> types = new HashMap();
> }
> public void setLexer(MjcLexer lexer) {
> this.lexer = lexer;
> }
> public void setType(int type) {
> this.type = type;
> }
> private IElementType addType(int type) {
> iType = new IElementType(Integer.toString(type),
> MjcSupportLoader.MJCFILE.getLanguage());
> types.put(new Integer(type), iType);
> return (IElementType)(types.get(new Integer(type)));
> }
> private IElementType getType(int type) {
> if(types.containsKey(new Integer(type))) {
> return (IElementType)types.get(new Integer(type));
> } else {
> return addType(type);
> }
> }
> public MjcLexer getLexer() {
> return lexer;
> }
> public final void start(char[] buffer) {
> start(((char[]) buffer), 0, ((char[]) buffer).length);
> }
> public final void start(char[] buffer, int startOffset, int
> endOffset) {
> start(((char[]) buffer), startOffset, endOffset, 0);
> }
> public void start(char[] buffer, int startOffset, int endOffset, int
> initialState) {
> String temp = "";
> for (int i = startOffset; i < endOffset; i++) {
> temp += buffer+;
> }
> lexer = new MjcLexer(handler, new JavaSource(handler, "MJC Source",
> new StringReader(temp)));
> type  = lexer.nextToken();
> }
> public int getState() {
> return 0;
> }
> public int getLastState() {
> return 0;
> }
> public IElementType getTokenType() {
> if(type == 0) {
> return null;
> }
> return getType(type);
> }
> public int getTokenStart() {
> return lexer.getPos().getColumn();
> }
> public int getTokenEnd() {
> int posCol = lexer.getPos().getColumn();
> String lex = lexer.getLexeme();
> if (lex == null) {
> return posCol + 1;
> }
> return posCol + lex.length();
> }
> public void advance() {
> type = lexer.nextToken();
> }
> public char[] getBuffer() {
> String lex = lexer.getLexeme();
> if (lex == null) {
> return new char[]{0};
> }
> return lexer.getLexeme().toCharArray();
> }
> public int getBufferEnd() {
> String lex = lexer.getLexeme();
> if (lex == null) {
> return 0;
> }
> return lexer.getLexeme().length();
> }
> public int getSmartUpdateShift() {
> return 0;  //To change body of implemented methods use File |
> Settings | File Templates.
> }
> public Object clone() {
> try {
> return super.clone();
> } catch (CloneNotSupportedException e) {
> return null;
> }
> }
> }
> ]]>

Maxim,
Thanks for your help. Hopefully, having this discussion here will also help other people with some of these problems. I'll also update the wiki if that's still the best place to store this kind of info?

adam

Yep, wiki is still the place. We're going to setup Confluence site for the
community but not sure yet how to organize the stuff so it won't go wiki's
steps to chaos.
-

Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

Maxim,
Thanks for your help. Hopefully, having this discussion here will
also help other people with some of these problems. I'll also update
the wiki if that's still the best place to store this kind of info?
adam

Maxim

.. Confluence ... how to organize the stuff so it won't go wiki's
steps to chaos.

Just curious: why do you think Confluence is better than Wiki? The same
people will produce the same chaos. In a more structured way, though.


Alain

Any ideas?
-

Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

Maxim

>> .. Confluence ... how to organize the stuff so it won't go wiki's
>> steps to chaos.
>>

Just curious: why do you think Confluence is better than Wiki? The
same people will produce the same chaos. In a more structured way,
though.

Alain

Maxim

Any ideas?

Yep :)
For the openApi:

1/
Write a webApp with Fabrique that looks and works like the java Almanac,
but authorizes collaboration/commenting.
http://javaalmanac.com/

2/
Install an annotatable/ted version of the open javadoc (looks like the
javadoc, but users can comment on individual items).


Not having the 2 above with either meet silence - no participation, no
info -, or have people jump through loops to try and give the same kind
of service.

Alain

Any ideas?

Do not let the comunity(or at least the most of them) to define the structure
- only the content. You control the structure with predefined themes/threads/structure,
and the comunity can add content only in those predefined places. Than you
do an active filtering, removing spam/bloat , and moving "old content" to
other structures

You can achieve this very simple with a small application written with Fabrique.

This way you give to the Fabrique team to make as sample a really useful
application, not those "too theoretical" demos that come at the moment :).
Even more, you can than use that application for yourself to ensure that
the all the product comunity pages have the same structure. If you look at
those from Fabrique you will see a lot of varations, just like the comunity
pages from the WIKI :)

The publishing framework can be Confluence too, but IMHO that's too complicated.
Confluence and Wikis are just "too democratic" and require too much discipline.
Considering how big is the comunity,it's utopic to think that this "discipline
goal" will be ever achieved.

Ahmed.

Yep :)
For the openApi:
1/
Write a webApp with Fabrique that looks and works like the java
Almanac,
but authorizes collaboration/commenting.
http://javaalmanac.com/

I proposed this a while ago, but no one showed interest.
I don't know why didn't they made it before, since Fabrique is pretty stable
for such small applications. This way they would have some REAL demos too
for Fabrique, not just that useless "librarian" :).

Ahmed.

Do not let the comunity(or at least the most of them) to define the
structure - only the content. You control the structure with
predefined themes/threads/structure,

I have a personal problem with this kind of
rigid-structure-before-contents reflex: it is to text what BDUF is too
code: artificially imposed, often bloated and always sub-optimal.




Alain

I have a personal problem with this kind of
rigid-structure-before-contents reflex: it is to text what BDUF is too
code: artificially imposed, often bloated and always sub-optimal.

Than you get the soup that is actually on the WIKI.

Besides, very active EAPers could get more rights (maybe to define new structures/templates),
so you can choose what is "optimal".

Ahmed.

Ahmed

Than you get the soup that is actually on the WIKI.

>

Refactoring doesn't work only with code. JB should have devoted - part
of - a person to maintain the wiki. They pay graphist, system manager;
why not a dedicated "community manager", that would be responsible of
tuning the wiki.

Besides, very active EAPers could get more rights (maybe to define new
structures/templates), so you can choose what is "optimal".

>

Again, "structure/templates"... and you end-up with
2 lines of info in a new and short page
that gets hidden between the mandatory/automatic
toolbar +
header +
TOC +
<![CDATA[]]> +
footer + revision + author + bread crumbs + ....

It reminds me too much of people who keep asking for a tool to document
their getters and setters. I know, I'm addicted to lean interface.


Alain