Remaining Language API questions

Hello, over the past week or two I've posted some Language API
questions. Some of you from JB and from the community have helped me out
but there are a few questions that I'd like very much answers to, which
have not been answered. I hope this re-post will make the questions
visible again so someone can answer.

1. "Language API: how to embed languages within each other?"

I want to embed HTML, CSS, and Java into a custom XML language. I also
want to embed Java into another custom language. Is this possible at all?

2. "Re: Language API: custom XML file formats"

Dmitry Jemerov answered part of my question, but it still remains: is
there a way to define custom XML formats with custom completion of tag
names, attributes, and contents?

3. "Find Usages only partly works"

Eugene, Dmitry, and Maxim Mossienko helped me out with this, and the
situation has improved for it, but find usages and refactor->rename
still doesn't fully work for my language. When I refactor->rename, the
item is renamed, but references are not changed, etc. Will someone go
over the steps required to support these features, including who should
implement what, and what should return what?

These are all of my questions for now, I hope someone can help me with
these problems.

31 comments
Comment actions Permalink

Hello Keith,

KL> 1. "Language API: how to embed languages within each other?"
KL>
KL> I want to embed HTML, CSS, and Java into a custom XML language. I
KL> also want to embed Java into another custom language. Is this
KL> possible at all?

Yes, this is possible. For the text range in your custom language file which
is in another language, your lexer should return a single token with element
type derived from com.intellij.psi.tree.IChameleonElementType. The specific
chameleon element types for standard languages are not in the OpenAPI (for
example, CSS_STYLESHEET in com.intellij.psi.css.impl.CssElementTypes, or
STATEMENTS in com.intellij.psi.impl.source.tree.JavaElementType), but you
could try using them nevertheless.

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Hello Keith,

KL> 2. "Re: Language API: custom XML file formats"
KL>
KL> Dmitry Jemerov answered part of my question, but it still remains:
KL> is there a way to define custom XML formats with custom completion
KL> of tag names, attributes, and contents?

Unfortunately none of the classes required to implement this are currently
in OpenAPI. As a hint, similar functionality is implemented by the JavaScriptCompletionData
class in the (not open-source) JsHtmlBridge plugin.

--
Dmitry Jemerov
Omea Project Leader
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Keith Lea wrote:

3. "Find Usages only partly works"

Eugene, Dmitry, and Maxim Mossienko helped me out with this, and the
situation has improved for it, but find usages and refactor->rename
still doesn't fully work for my language. When I refactor->rename, the
item is renamed, but references are not changed, etc. Will someone go
over the steps required to support these features, including who should
implement what, and what should return what?


Keith, have you implemented PsiReference.handleElementRename()? The code you posted some
time ago didn't have this. I'm not totally sure about the exactly correct process, but I
believe your reference is responsible for renaming the element it refers to. This doesn't
happen automatically.

int x = 0; <- rename x to y
x = 1; <- PsiReference.handleElementRename("y") should change "x" to "y"

Sascha

PS: Dmitry, what's the difference between handleElementRename() and bindToElement()? When
is the latter one called?

0
Comment actions Permalink

Hello Sascha,

SW> PS: Dmitry, what's the difference between handleElementRename() and
SW> bindToElement()? When is the latter one called?

bindToElement() is called to change a reference so that it becomes a reference
to the specified element. This may involve more than simply changing the
text of the reference - for example, the reference may need to use the full-qualified
name of the referred element.

This method is used by many refactoring and intention actions. For example,
the "Create Class from New" intention uses it to bind the (invalid) reference
on which the intention was called to the newly created class.

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Okay, we're close, references almost all works now. Refactor->rename
works, but only when the cursor is in a reference, not when the cursor
is in a PsiNamedElement which references point to. I still can't Find
Usages when the cursor is in the PsiNamedElement either. What do you
think is the problem with this, Sacha and Dmitry?

Sascha Weinreuter wrote:

Keith Lea wrote:

>>3. "Find Usages only partly works"
>>
>>Eugene, Dmitry, and Maxim Mossienko helped me out with this, and the
>>situation has improved for it, but find usages and refactor->rename
>>still doesn't fully work for my language. When I refactor->rename, the
>>item is renamed, but references are not changed, etc. Will someone go
>>over the steps required to support these features, including who should
>>implement what, and what should return what?


Keith, have you implemented PsiReference.handleElementRename()? The code you posted some
time ago didn't have this. I'm not totally sure about the exactly correct process, but I
believe your reference is responsible for renaming the element it refers to. This doesn't
happen automatically.

int x = 0; <- rename x to y
x = 1; <- PsiReference.handleElementRename("y") should change "x" to "y"

Sascha

PS: Dmitry, what's the difference between handleElementRename() and bindToElement()? When
is the latter one called?

0
Comment actions Permalink

Dmitry Jemerov (JetBrains) wrote:

Hello Keith,

KL> 1. "Language API: how to embed languages within each other?"
KL> KL> I want to embed HTML, CSS, and Java into a custom XML language. I
KL> also want to embed Java into another custom language. Is this
KL> possible at all?

Yes, this is possible. For the text range in your custom language file
which is in another language, your lexer should return a single token
with element type derived from
com.intellij.psi.tree.IChameleonElementType. The specific chameleon
element types for standard languages are not in the OpenAPI (for
example, CSS_STYLESHEET in com.intellij.psi.css.impl.CssElementTypes, or
STATEMENTS in com.intellij.psi.impl.source.tree.JavaElementType), but
you could try using them nevertheless.


Great, I will try this. Is there a facility to insert custom language
into already existing language? For example, the web framework I use,
RIFE, uses a template system based on HTML comments. I'd like to provide
highlighting & completion for certain-formatted HTML comments, in all
HTML/XHTML files. How can I do this?

0
Comment actions Permalink

Hello Keith,

KL> Great, I will try this. Is there a facility to insert custom
KL> language into already existing language? For example, the web
KL> framework I use, RIFE, uses a template system based on HTML
KL> comments. I'd like to provide highlighting & completion for
KL> certain-formatted HTML comments, in all HTML/XHTML files. How can I
KL> do this?

There is currently no OpenAPI support for this. You could try to build something
based on overriding com.intellij.lexer.HtmlLexer.getTokenType() and creating
similar chameleon tokens for the special HTML comments, but I'm afraid this
is hardcore hackery for which we can't really provide support. :)

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Dmitry Jemerov (JetBrains) wrote:

Hello Keith,

KL> 2. "Re: Language API: custom XML file formats"
KL> KL> Dmitry Jemerov answered part of my question, but it still remains:
KL> is there a way to define custom XML formats with custom completion
KL> of tag names, attributes, and contents?

Unfortunately none of the classes required to implement this are
currently in OpenAPI. As a hint, similar functionality is implemented by
the JavaScriptCompletionData class in the (not open-source) JsHtmlBridge
plugin.


I've been looking through jad-decompiled JsHtmlBridge but it doesn't
reveal a lot to me at first. Will these classes be released as
open-source some time?

0
Comment actions Permalink

Hello Keith,

>> KL> 2. "Re: Language API: custom XML file formats"
>> KL> KL> Dmitry Jemerov answered part of my question, but it still
>> remains:
>> KL> is there a way to define custom XML formats with custom
>> completion
>> KL> of tag names, attributes, and contents?
>> Unfortunately none of the classes required to implement this are
>> currently in OpenAPI. As a hint, similar functionality is implemented
>> by the JavaScriptCompletionData class in the (not open-source)
>> JsHtmlBridge plugin.
>>
KL> I've been looking through jad-decompiled JsHtmlBridge but it doesn't
KL> reveal a lot to me at first. Will these classes be released as
KL> open-source some time?

We don't plan to release as open-source plugins which use non-OpenAPI classes.

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Hello Keith,

KL> Okay, we're close, references almost all works now. Refactor->rename
KL> works, but only when the cursor is in a reference, not when the
KL> cursor is in a PsiNamedElement which references point to. I still
KL> can't Find Usages when the cursor is in the PsiNamedElement either.
KL> What do you think is the problem with this, Sacha and Dmitry?

One likely reason: if the name identifier within the PsiNamedElement does
not start at the beginning of the element text range, you need to override
PsiElement.getTextOffset() and return the offset of the name identifier within
the element. See JSFunctionImpl.getTextOffset() in the JavaScript plugin
for an example.

If this doesn't help, could you post your code so that we could look at it
again?

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Dmitry Jemerov (JetBrains) wrote:

Hello Keith,

KL> Okay, we're close, references almost all works now. Refactor->rename
KL> works, but only when the cursor is in a reference, not when the
KL> cursor is in a PsiNamedElement which references point to. I still
KL> can't Find Usages when the cursor is in the PsiNamedElement either.
KL> What do you think is the problem with this, Sacha and Dmitry?

One likely reason: if the name identifier within the PsiNamedElement
does not start at the beginning of the element text range, you need to
override PsiElement.getTextOffset() and return the offset of the name
identifier within the element. See JSFunctionImpl.getTextOffset() in the
JavaScript plugin for an example.

If this doesn't help, could you post your code so that we could look at
it again?


I implement getTextOffset correctly, as far as I know. Here is some code:


public class ProtoMessageTypeReference extends ASTWrapperPsiElement
implements ProtoElement, PsiReference {
public ProtoMessageTypeReference(ASTNode astNode) {super(astNode);}

public String getReferencedName() {
ASTNode node = getIdentifierNode();
return node == null ? "" : node.getText();
}

private ASTNode getIdentifierNode() {
return getNode().findChildByType(ProtoTokenTypes.IDENTIFIER);
}

public PsiReference getReference() {
return this;
}

public PsiElement getElement() {
return this;
}

public TextRange getRangeInElement() {
return new TextRange(0, getTextLength());
}

@Nullable
public PsiElement resolve() {
ProtoFile protoFile = PsiTreeUtil.getParentOfType(this,
ProtoFile.class);
if (protoFile == null) return null;
return protoFile.getMessageDefinition(getReferencedName());
}

public String getCanonicalText() {
return getReferencedName();
}

public PsiElement handleElementRename(String newElementName)
throws IncorrectOperationException {
setReferencedName(newElementName);
return this;
}

public PsiElement bindToElement(PsiElement element)
throws IncorrectOperationException {
ProtoMessageDefinition def = (ProtoMessageDefinition) element;

setReferencedName(def.getName());
return this;
}

private void setReferencedName(String name) {
ASTNode newNode = ProtoChangeTools.createIdentifierFromText(
getProject(), name);
ASTNode oldNode = getIdentifierNode();
oldNode.getTreeParent().replaceChild(oldNode, newNode);
}

public boolean isReferenceTo(PsiElement element) {
if (element instanceof ProtoMessageDefinition) {
ProtoMessageDefinition msg = (ProtoMessageDefinition) element;
String name = msg.getName();
return name != null && name.equals(getReferencedName());
} else {
return false;
}
}

public Object[] getVariants() {
ProtoFile protoFile = PsiTreeUtil.getParentOfType(this,
ProtoFile.class);
if (protoFile == null) return new Object[0];
List]]> defs =
protoFile.getMessageDefinitions();
return defs.toArray();
}

public boolean isSoft() {
// ??
return false;
}
}

ProtoMessageDefinition is a subclass of:

public class ProtoNamedElementImpl extends ASTWrapperPsiElement implements
ProtoElement, PsiNamedElement {
public ProtoNamedElementImpl(com.intellij.lang.ASTNode astNode) {
super(astNode);
}

public ProtoNameElement getNameElement() {
return ProtoTools.findDirectChildOfType(this,
ProtoNameElement.class);
}

public int getTextOffset() {
ProtoNameElement nameElement = getNameElement();
return nameElement == null ? super.getTextOffset() :
nameElement.getTextOffset();
}

public @Nullable String getName() {
ProtoNameElement nameElement = getNameElement();
return nameElement == null ? null : nameElement.getName();
}

public PsiElement setName(String name) throws
IncorrectOperationException {
getNameElement().setName(name);
return this;
}

public ItemPresentation getPresentation() {
return new ProtoItemPresentation(this);
}
}

0
Comment actions Permalink

Hello Keith,

>> KL> Okay, we're close, references almost all works now.
>> Refactor->rename
>> KL> works, but only when the cursor is in a reference, not when the
>> KL> cursor is in a PsiNamedElement which references point to. I still
>> KL> can't Find Usages when the cursor is in the PsiNamedElement
>> either.
>> KL> What do you think is the problem with this, Sacha and Dmitry?
>> One likely reason: if the name identifier within the PsiNamedElement
>> does not start at the beginning of the element text range, you need
>> to override PsiElement.getTextOffset() and return the offset of the
>> name identifier within the element. See
>> JSFunctionImpl.getTextOffset() in the JavaScript plugin for an
>> example.
>>
>> If this doesn't help, could you post your code so that we could look
>> at it again?
>>
KL> I implement getTextOffset correctly, as far as I know. Here is some
KL> code:

In fact, one of your guesses in the older thread related to this problem
was correct: for Rename/Find Usages on a declaration to work, the PsiNamedElement
must be the direct parent of the identifier node. As far as I understand,
this isn't the case with your implementation now.

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Dmitry Jemerov (JetBrains) wrote:

Hello Keith,

KL> 1. "Language API: how to embed languages within each other?"
KL> KL> I want to embed HTML, CSS, and Java into a custom XML language. I
KL> also want to embed Java into another custom language. Is this
KL> possible at all?

Yes, this is possible. For the text range in your custom language file
which is in another language, your lexer should return a single token
with element type derived from
com.intellij.psi.tree.IChameleonElementType. The specific chameleon
element types for standard languages are not in the OpenAPI (for
example, CSS_STYLESHEET in com.intellij.psi.css.impl.CssElementTypes, or
STATEMENTS in com.intellij.psi.impl.source.tree.JavaElementType), but
you could try using them nevertheless.


I'd like to embed an import list into my custom language. I return
JavaElementType.IMPORT_LIST from my lexer for such content, and in my
parser, I skip over the resulting IMPORT_LIST_TEXT identifier that is
passed. However, I get this exception:
Index: 4, Size: 4
java.lang.IndexOutOfBoundsException: Index: 4, Size: 4
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at com.intellij.lang.impl.PsiBuilderImpl.a(PsiBuilderImpl.java:4)
at
com.intellij.lang.impl.PsiBuilderImpl.getCurrentToken(PsiBuilderImpl.java:36)
at com.intellij.lang.impl.PsiBuilderImpl.eof(PsiBuilderImpl.java:56)
at net.kano.genproto.lex.ProtoParser.parse(ProtoParser.java:61)
at com.intellij.extapi.psi.PsiFileBase.a(PsiFileBase.java:16)
at com.intellij.extapi.psi.PsiFileBase.(PsiFileBase.java:4) at net.kano.genproto.psi.ProtoFile.]]>(ProtoFile.java:57)
at
net.kano.genproto.lex.ProtoParserDefinition.createFile(ProtoParserDefinition.java:200)
at
com.intellij.psi.impl.PsiElementFactoryImpl.createFileFromText(PsiElementFactoryImpl.java:81)
at
com.intellij.psi.impl.source.text.BlockSupportImpl.a(BlockSupportImpl.java:104)
at
com.intellij.psi.impl.source.text.BlockSupportImpl.reparseRangeInternal(BlockSupportImpl.java:60)
at
com.intellij.psi.impl.source.text.BlockSupportImpl.reparseRange(BlockSupportImpl.java:16)
at
com.intellij.psi.impl.PsiDocumentManagerImpl.commit(PsiDocumentManagerImpl.java:107)
at
com.intellij.psi.impl.PsiDocumentManagerImpl$1.run(PsiDocumentManagerImpl.java:5)
at
com.intellij.openapi.application.impl.ApplicationImpl.runWriteAction(ApplicationImpl.java:438)
at
com.intellij.psi.impl.PsiDocumentManagerImpl.commitDocument(PsiDocumentManagerImpl.java:203)
at
com.intellij.psi.impl.PsiDocumentManagerImpl.commitAllDocuments(PsiDocumentManagerImpl.java:99)
at
com.intellij.codeInsight.daemon.impl.TextEditorBackgroundHighlighter.a(TextEditorBackgroundHighlighter.java:30)
at
com.intellij.codeInsight.daemon.impl.TextEditorBackgroundHighlighter.createPassesForEditor(TextEditorBackgroundHighlighter.java:10)
at
com.intellij.codeInsight.daemon.impl.TextEditorBackgroundHighlighter.createPassesForEditor(TextEditorBackgroundHighlighter.java:47)
at
com.intellij.codeInsight.daemon.impl.DaemonCodeAnalyzerImpl.a(DaemonCodeAnalyzerImpl.java:140)
at
com.intellij.codeInsight.daemon.impl.DaemonCodeAnalyzerImpl.access$2000(DaemonCodeAnalyzerImpl.java:194)
at
com.intellij.codeInsight.daemon.impl.DaemonCodeAnalyzerImpl$8$1UpdateEditorRunnable.run(DaemonCodeAnalyzerImpl.java:5)
at
com.intellij.codeInsight.daemon.impl.DaemonCodeAnalyzerImpl$8.run(DaemonCodeAnalyzerImpl.java:3)
at com.intellij.util.Alarm$1.run(Alarm.java:96)
at com.intellij.util.Alarm$MyThread$1.run(Alarm.java:238)
at
com.intellij.openapi.application.impl.LaterInvocatorEx$FlushQueue.run(LaterInvocatorEx.java:16)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:461)
at
com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:65)
at com.intellij.ide.IdeEventQueue.a(IdeEventQueue.java:56)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:160)
at
java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:267)
at
java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:196)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:190)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:182)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)

Am I doing something wrong?

0
Comment actions Permalink

Hello Keith,

KL> I'd like to embed an import list into my custom language. I return
KL> JavaElementType.IMPORT_LIST from my lexer for such content, and in my
KL> parser, I skip over the resulting IMPORT_LIST_TEXT identifier that is
KL> passed. However, I get this exception:
KL> Index: 4, Size: 4
KL> java.lang.IndexOutOfBoundsException: Index: 4, Size: 4
KL> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
KL> at java.util.ArrayList.get(ArrayList.java:322)
KL> at com.intellij.lang.impl.PsiBuilderImpl.a(PsiBuilderImpl.java:4)

KL> Am I doing something wrong?

Yes and no. :) You're calling PsiBuilder.advanceLexer() too many times,
causing it to move beyond the end of file. The exception, however, is a bug
in IDEA which will be fixed in 5.0.1 - it will simply return true from the
eof() check in this case.

--
Dmitry Jemerov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

Dmitry Jemerov (JetBrains) wrote:

Hello Keith,

KL> I'd like to embed an import list into my custom language. I return
KL> JavaElementType.IMPORT_LIST from my lexer for such content, and in my
KL> parser, I skip over the resulting IMPORT_LIST_TEXT identifier that is
KL> passed. However, I get this exception:
KL> Index: 4, Size: 4
KL> java.lang.IndexOutOfBoundsException: Index: 4, Size: 4
KL> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
KL> at java.util.ArrayList.get(ArrayList.java:322)
KL> at com.intellij.lang.impl.PsiBuilderImpl.a(PsiBuilderImpl.java:4)

KL> Am I doing something wrong?

Yes and no. :) You're calling PsiBuilder.advanceLexer() too many times,
causing it to move beyond the end of file. The exception, however, is a
bug in IDEA which will be fixed in 5.0.1 - it will simply return true
from the eof() check in this case.


Are you sure this is the only cause of the exception? I am debugging and
I don't see why or when this exception is thrown - when I step through
my parser, everything completes successfully. Here are parts of my .flex
file:

JAVA_LITERAL="java "
IDENTIFIER=[:jletter:] *
TO_END_OF_LINE=[^\r\n]*

%state EOL

%%

]]> { yybegin(YYINITIAL); return
ProtoTokenTypes.JAVA_LITERAL; }

  1. ^-- JAVA_LITERAL is actually IMPORT_LIST

+ { return ProtoTokenTypes.WHITE_SPACE; }
{ yybegin(EOL); return
ProtoTokenTypes.LANGUAGE_LITERAL; }

My parser code simply does this:

while (!builder.eof()) {
if (builder.getTokenType() ==
ProtoTokenTypes.LANGUAGE_LITERAL) {
PsiBuilder.Marker marker = builder.mark();
parseNextTokenAsKeyword(builder); // simply advances lexer
builder.advanceLexer(); // skip java token
marker.done(ProtoElementTypes.LANGUAGE_LITERAL);
continue;
}
// other parsing...
}

Sometimes, the import statement works - it's highlighted correctly and
completion works. But usually I get that exception, and the first token
after an import statement seems to be skipped completely, even though
when I step through, I see that I'm not advancing over it.

Do you think you see what's wrong? I hope I'm not taking too much of
your time for a dumb problem. I hope once this is resolved, others can
learn from this by searching the forums.

0
Comment actions Permalink

Dmitry Jemerov (JetBrains) wrote:

Hello Keith,

>>> KL> Okay, we're close, references almost all works now.
>>> Refactor->rename
>>> KL> works, but only when the cursor is in a reference, not when the
>>> KL> cursor is in a PsiNamedElement which references point to. I still
>>> KL> can't Find Usages when the cursor is in the PsiNamedElement
>>> either.
>>> KL> What do you think is the problem with this, Sacha and Dmitry?
>>> One likely reason: if the name identifier within the PsiNamedElement
>>> does not start at the beginning of the element text range, you need
>>> to override PsiElement.getTextOffset() and return the offset of the
>>> name identifier within the element. See
>>> JSFunctionImpl.getTextOffset() in the JavaScript plugin for an
>>> example.
>>>
>>> If this doesn't help, could you post your code so that we could look
>>> at it again?
>>>

KL> I implement getTextOffset correctly, as far as I know. Here is some
KL> code:

In fact, one of your guesses in the older thread related to this problem
was correct: for Rename/Find Usages on a declaration to work, the
PsiNamedElement must be the direct parent of the identifier node. As far
as I understand, this isn't the case with your implementation now.


The name is a custom PsiElement called ProtoNameElement, which is direct
child of ProtoMessageDefinition. However, there may be a PsiLeafElement
or whatever, for the actual AST node. You think this is the problem?
I'll look into it.

0
Comment actions Permalink

Dmitry Jemerov (JetBrains) wrote:

Hello Keith,

>>> KL> Okay, we're close, references almost all works now.
>>> Refactor->rename
>>> KL> works, but only when the cursor is in a reference, not when the
>>> KL> cursor is in a PsiNamedElement which references point to. I still
>>> KL> can't Find Usages when the cursor is in the PsiNamedElement
>>> either.
>>> KL> What do you think is the problem with this, Sacha and Dmitry?
>>> One likely reason: if the name identifier within the PsiNamedElement
>>> does not start at the beginning of the element text range, you need
>>> to override PsiElement.getTextOffset() and return the offset of the
>>> name identifier within the element. See
>>> JSFunctionImpl.getTextOffset() in the JavaScript plugin for an
>>> example.
>>>
>>> If this doesn't help, could you post your code so that we could look
>>> at it again?
>>>

KL> I implement getTextOffset correctly, as far as I know. Here is some
KL> code:

In fact, one of your guesses in the older thread related to this problem
was correct: for Rename/Find Usages on a declaration to work, the
PsiNamedElement must be the direct parent of the identifier node. As far
as I understand, this isn't the case with your implementation now.

By the way, my parser works fine as long as there aren't Java fragments
in the file.

0
Comment actions Permalink

The bug is somewhat more funny actually.

It seem you're calling builder.advanceLexer() a few times without asking
anything else from the builder in between like checking for eof or getTokenType().
It is clearly bug in IDEA (fixed) but may also cause bugs in your parser
if there's no more lexems could be retreiven in reality.

-


Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"


0
Comment actions Permalink

It's also probably a bug in my code. I now call builder.eof() before
advanceLexer and the embedded Java works great. Thanks Max & Dmitry! I
have some follow-up questions that I'll start in a new thread, to make
it easy for future language API users to search for it.

Maxim Shafirov (JetBrains) wrote:

The bug is somewhat more funny actually.

It seem you're calling builder.advanceLexer() a few times without asking
anything else from the builder in between like checking for eof or
getTokenType(). It is clearly bug in IDEA (fixed) but may also cause
bugs in your parser if there's no more lexems could be retreiven in
reality.

-------------------
Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

0
Comment actions Permalink

I've filed http://jetbrains.net/jira/browse/IDEA-4304 about this behavior.

Dmitry Jemerov (JetBrains) wrote:

Hello Keith,

>>> KL> Okay, we're close, references almost all works now.
>>> Refactor->rename
>>> KL> works, but only when the cursor is in a reference, not when the
>>> KL> cursor is in a PsiNamedElement which references point to. I still
>>> KL> can't Find Usages when the cursor is in the PsiNamedElement
>>> either.
>>> KL> What do you think is the problem with this, Sacha and Dmitry?
>>> One likely reason: if the name identifier within the PsiNamedElement
>>> does not start at the beginning of the element text range, you need
>>> to override PsiElement.getTextOffset() and return the offset of the
>>> name identifier within the element. See
>>> JSFunctionImpl.getTextOffset() in the JavaScript plugin for an
>>> example.
>>>
>>> If this doesn't help, could you post your code so that we could look
>>> at it again?
>>>

KL> I implement getTextOffset correctly, as far as I know. Here is some
KL> code:

In fact, one of your guesses in the older thread related to this problem
was correct: for Rename/Find Usages on a declaration to work, the
PsiNamedElement must be the direct parent of the identifier node. As far
as I understand, this isn't the case with your implementation now.

0
Comment actions Permalink

Dmitry, I am using CompletionUtil and I'm not having any success. I have a
breakpoint at every point I can find to set a breakpoint, and none of it is
being called. I know this is not published API, but right now completion in my
language is a big annoyance so I'd like to implement it. I hope you can look at
my code and tell me why none of it is being called.

I call this inside ApplicationComponent.initComponent (it doesn't work inside
ProjectComponent either):

CompletionUtil.registerCompletionData(PROTO_FILE_TYPE, new
CompletionData() {
{
CompletionVariant eqVariant = new CompletionVariant(
new ElementFilter() {
public boolean isAcceptable(Object object,
PsiElement psiElement) {
return true;
}

public boolean isClassAcceptable(Class aClass) {
return true;
}
});
eqVariant.includeScopeClass(LeafPsiElement.class);
eqVariant.setInsertHandler(new InsertHandler() {
public void handleInsert(CompletionContext
completionContext, int i,
LookupData lookupData, LookupItem lookupItem,
boolean b, char c) {
System.out.println("hi");
}
});
eqVariant.addCompletionFilterOnElement(new ElementFilter() {
public boolean isAcceptable(Object object, PsiElement
psiElement) {
return true;
}

public boolean isClassAcceptable(Class aClass) {
return true;
}
});
eqVariant.addCompletionFilter(new ElementFilter() {
public boolean isAcceptable(Object object, PsiElement
psiElement) {
return true;
}

public boolean isClassAcceptable(Class aClass) {
return true;
}
});
eqVariant.addCompletion("hi");
eqVariant.addCompletion(new ContextGetter() {
public Object[] get(PsiElement psiElement,
CompletionContext completionContext) {
return new Object[0];
}
}, 0);
registerVariant(eqVariant);
}
public String findPrefix(PsiElement psiElement, int i) {
return WordCompletionData.findPrefixSimple(psiElement, i);
}
});

Each place you see a return statement, I have a breakpoint, and none of the
breakpoints are being hit, no matter where I press Ctrl+Space.

Dmitry Jemerov (JetBrains) wrote:

Hello Keith,

KL> 2. "Re: Language API: custom XML file formats"
KL> KL> Dmitry Jemerov answered part of my question, but it still remains:
KL> is there a way to define custom XML formats with custom completion
KL> of tag names, attributes, and contents?

Unfortunately none of the classes required to implement this are
currently in OpenAPI. As a hint, similar functionality is implemented by
the JavaScriptCompletionData class in the (not open-source) JsHtmlBridge
plugin.

0
Comment actions Permalink

Keith, try using a non-anonymous class as your CompletionData implementation and
do the following in its constructor:

declareFinalScope(.class); final CompletionVariant variant = new CompletionVariant(); variant.includeScopeClass(.class, true); variant.addCompletionFilterOnElement(TrueFilter.INSTANCE); variant.addCompletion(...) registerVariant(variant); where is the class all your PSI classes are derived from, and ]]> is the filter to decides whether the variant applies to a certain
position. You can use TrueFilter.INSTANCE for testing.

This works like a charm for me.

Sascha

0
Comment actions Permalink

Sascha, it doesn't work at all for me. I hope you will look at my code and see
if you see anything missing.

public class ProtoProjectComponent implements ProjectComponent {
public void projectOpened() {
CompletionUtil.registerCompletionData(PROTO_FILE_TYPE,
new MyCompletionData());
}

public void projectClosed() {
}

public String getComponentName() {
return "Protocol Buffer Loader";
}

public void initComponent() {
}

public void disposeComponent() {
}

private static class MyCompletionData extends CompletionData {
{
CompletionVariant eqVariant = new
CompletionVariant(TrueFilter.INSTANCE);
eqVariant.includeScopeClass(ProtoElement.class);
eqVariant.addCompletionFilterOnElement(TrueFilter.INSTANCE);
eqVariant.addCompletion("hi");
registerVariant(eqVariant);
}

public String findPrefix(PsiElement psiElement, int i) {
return WordCompletionData.findPrefixSimple(psiElement, i);
}
}
}

Sascha Weinreuter wrote:

Keith, try using a non-anonymous class as your CompletionData implementation and
do the following in its constructor:

declareFinalScope(<YourPsiBaseClass>.class);

final CompletionVariant variant = new CompletionVariant(<elementFilter>);

variant.includeScopeClass(<YourPsiBaseClass>.class, true);
variant.addCompletionFilterOnElement(TrueFilter.INSTANCE);
variant.addCompletion(...)

registerVariant(variant);

where <YourPsiBaseClass> is the class all your PSI classes are derived from, and
<elementFilter> is the filter to decides whether the variant applies to a certain
position. You can use TrueFilter.INSTANCE for testing.

This works like a charm for me.

Sascha

0
Comment actions Permalink

Keith, I don't use the addCompletion(String) method, but addCompletion(ContextGetter). I'm
also not entirely sure what the difference is between addCompletion(String) and
addCompletion(String, int). You could try using the latter one with 0 for the int argument.

The only other difference I see so far is that you call includeScopeClass(Class), not
includeScopeClass(Class, boolean). Try using this one and specify true as the argument.

If that still doesn't work, try to use LeafPsiElement.class as the scope, maybe there's
something wrong with your ProtoElement class not being recognized correctly.

HTH,
Sascha

0
Comment actions Permalink

Do you perform initialisation in ProjectComponent? Is it in initComponent or
projectOpened? I tried passing 0 and true where you said, and I tried
LeafPsiElement, and none of it works. Is there any other special setup that is
required for my language to support this kind of completion, that you know of?

Sascha Weinreuter wrote:

Keith, I don't use the addCompletion(String) method, but addCompletion(ContextGetter). I'm
also not entirely sure what the difference is between addCompletion(String) and
addCompletion(String, int). You could try using the latter one with 0 for the int argument.

The only other difference I see so far is that you call includeScopeClass(Class), not
includeScopeClass(Class, boolean). Try using this one and specify true as the argument.

If that still doesn't work, try to use LeafPsiElement.class as the scope, maybe there's
something wrong with your ProtoElement class not being recognized correctly.

HTH,
Sascha

0
Comment actions Permalink

Keith Lea wrote:

Do you perform initialisation in ProjectComponent? Is it in
initComponent or projectOpened?


I have used this in a ProjectComponent.projectOpened() and it worked. I'm also registering
the completion data for my "main" file type from a static initializer of the my FileType
implementation which works fine as well. I don't think it matters where/when the data is
registered, as long it is done before you need it. Since the registration process seems to
be Project-independent, I'd even say a ProjectComponent is the rather wrong place to do it.

Is there any other special setup that is required for my language to support this
kind of completion, that you know of?


No, not really. I just did the same as JsHtmlBridge does and had instant success.
Stupid question: Are you sure that CompletionUtil.registerCompletionData() is actually
called? Wouldn't be the first time that such weird things have a very simple cause ;)

Also try the following: In your ParserDefinition's createFile() methods, attach your
CompletionData instance to the created PsiFile by

psiFile.putUserData(CompletionUtil.ENFORCE_COMPLETION_DATA_KEY, data)

to see if it's a registration problem or not. If that doesn't work as well, I don't have
any better idea than overriding every possible method of CompletionData and the
CompletionVariant you're registering and setting breakpoints in them. If it does work, the
problem is most likely something with the registration process. Then try this and register
INSTANCE with the FileTypeManager to really make sure there's only one instance around:

class YourFileType extends LanguageFileType {
public static final YourFileType INSTANCE = new YourFileType();

private YourFileType() {
super(new YourFileTypeLanguage());
CompletionUtil.registerCompletionData(this, new YourCompletionData());
}

...
}

Sascha

0
Comment actions Permalink

Thanks for all your help Sashca, I've been changing and rewriting it and it
works for some reason now.

Sascha Weinreuter wrote:

Keith Lea wrote:

>> Do you perform initialisation in ProjectComponent? Is it in
>> initComponent or projectOpened?


I have used this in a ProjectComponent.projectOpened() and it worked.
I'm also registering the completion data for my "main" file type from a
static initializer of the my FileType implementation which works fine as
well. I don't think it matters where/when the data is registered, as
long it is done before you need it. Since the registration process seems
to be Project-independent, I'd even say a ProjectComponent is the rather
wrong place to do it.

>> Is there any other special setup that is required for my language to
>> support this kind of completion, that you know of?


No, not really. I just did the same as JsHtmlBridge does and had instant
success.
Stupid question: Are you sure that
CompletionUtil.registerCompletionData() is actually called? Wouldn't be
the first time that such weird things have a very simple cause ;)

Also try the following: In your ParserDefinition's createFile() methods,
attach your CompletionData instance to the created PsiFile by

psiFile.putUserData(CompletionUtil.ENFORCE_COMPLETION_DATA_KEY, data)

to see if it's a registration problem or not. If that doesn't work as
well, I don't have any better idea than overriding every possible method
of CompletionData and the CompletionVariant you're registering and
setting breakpoints in them. If it does work, the problem is most likely
something with the registration process. Then try this and register
INSTANCE with the FileTypeManager to really make sure there's only one
instance around:

class YourFileType extends LanguageFileType {
public static final YourFileType INSTANCE = new YourFileType();

private YourFileType() {
super(new YourFileTypeLanguage());
CompletionUtil.registerCompletionData(this, new
YourCompletionData());
}

...
}

Sascha

0
Comment actions Permalink

Is this still true that there is no OpenAPI to embed a custom language in a standard language? I have a proprietary scripting language for which I'd like to develop a language plugin. Its similar to JSP in that it is embedded in a text document using <%...%> syntax. Like JSP I'd like to maintain XML, HTML, CSS, JS support outside the scriptlet tags. Anyone done this?

0
Comment actions Permalink

Yup!
I got the same question.

I tried to parse everything outside my embedded code as some token type, then i made it a n InjectionHost and injected HTML into it but it was very poor mostly because those blocks were not connected between each other, so if you had your template code between opening and closing tags there was an error about "no closing tag".

I wonder how they would add RHTML support?

0
Comment actions Permalink


If I remember correctly, the plan is for RHTML to uses a new mechanism, which
would be available through OpenAPI for reuse in similar languages.
I think Mike Aizatsky mentioned this on the jetbrains-ruby-dev mailinglist
some time ago, so I don't know if that information is still current...

-tt


0

Please sign in to leave a comment.