How to allow Rename Refactoring to handle spaces in referencing Elements?

> two links refering to the same file will not get wordhighlighted when the caret is on one of them if the link has a space but will highlight if the link has no spaces. They will highlight if the link has non-standard identifier characters like @.

I may have misunderstood something but

if you are talking about in-place renaming, it appears that normally identifiers are only highlighted if RenameHandler found references to identifiers on the same page.

If identifiers in other pages are found, you get a rename popup dialog.

> when the file is renamed its refering links do not get renamed.

renaming file and renaming identifiers are separate operations.

> if the link has non-standard chars @ then hitting tab on a completion replaces text only to the @ and not the full node's text.

could you check in PsiViewer, is the link - one single token i.e.

XTypes.LINK

and not XTypes.LINK SPACE XTypes.LINK

?

I have wiki link elements in the custom language which contain WIKI_PAGE_REF elements, these are just user friendly references to files with the same name as the link text, but spaces are changed to dashes and .md extension appended.

[[sample document]] refers to sample-document.md, in the same directory as the containing file for the link.

[[NOTICE]] is refering to NOTICE.md in the same directory as the file where the link is located. I return a Reference to the PsiFile for NOTICE.md for the Element WIKI_PAGE_REF which is the 'NOTICE' part of [[NOTICE]]. Everything works fine. Doing refactor > rename on the link opens the file renaming dialog. Renaming the file will rename the link.

If I have: [[NOTICE]] [[NOTICE]] [[NOTICE]] in the file, putting the caret on one of them will hightlight all of them, which makes sence since they refer to the same object.

Moving the NOTICE.md file to another directory, lets say a subdirectory 'sub', (I remap the reference), also works to change all the refering links. With a catch, since the file is in another directory I remap the links in the ElementManipulator<MultiMarkdownWikiPageRef> extension, the links will be changed to [[sub/NOTICE]].

Up to this point life is great.

Except, now that the links have a /, putting the caret on one of them just highlights it but not the others that are pointing to the same file.

I can do refactor > rename on one of the links and the file rename dialog opens but renaming the file does not invoke any of the calls for refactoring. The links stay as they were and now I annotate them as missing because the original file is no longer there.

Same for code completion, as long as there are no funny characters TAB completion replaces the whole thing, even spaces are OK here. As soon as there is @, / completion replaces only up to the non-standard identifier character.

I tried making WIKI_PAGE_REF element both CODE and LITERAL, does not change the behaviour.

Effectively, something in the code in the IDEA is checking for the text contained in the PsiElement text range and not using the whole range without a check.

It is one single text range, I made sure of that.

Here is the pic of the psiviewer on one of the nodes, with the following node containing a space.


In the next one I renamed the [[NOTICE]] by doing refactor rename on one of the links and got the file rename dialog, where I renamed the file from NOTICE.md to NOTICE@.md, all links got updated and they still highlight when one of them is clicked on:



I can still do a refactor rename on the NOTICE@ links and get the file rename dialog, however, renaming the file will no longer invoke any of the refactoring calls. I can keep renaming the file as many times as I want and the links will follow suit as long as I don't include any characters that make refactoring stop recognizing the References from the nodes.

I've encountered this before but it was while playing with the 'SimplePlugin' which allows property keys to have escaped spaces 'key\ with \spaces' creates a key 'key with spaces' but then I could not get refactoring and completion working properly for these keys. Hoewever, it was just an experiment so I gave up.

Here I don't have the luxury. BTW, I was so eager to get this solved, I forgot to stop and thank you for the prompt reply. Sorry and thank you.

you are welcome Vladimir.

I just noticed that [[sample@document]]  is underlined as bad token. There is (!) icon in the top right corner too.

it seems that lexer does not permit / recognize sample@document as a valid token, no?

I actually do that in the Annotator to show the user that the document they refer to in the link does not exist, it is a bad link. [[sample document]] links to sample-document.md which exists and is not error highlighted. The [[sample@document]] I only changed by editing it to show that the Psi Tree has a single element WIKI_PAGE_REF that spans sample@document text range.

You can see that [[NOTICE@]] links all highlight together, they are referencing NOTICE@.md file and I can do a goto Reference (Command-B) on the Mac and the file will be opened that the link refers to.

The issue really is in the Rename refactoring, Move file refactoring (links with spaces or funny characters) and TAB completions for files that have funny characters only, spaces for TAB completions are ignored and the whole element is replaced.

> I can do refactor > rename on one of the links and the file rename dialog opens but renaming the file does not invoke any of the calls for refactoring.

>> renaming the file does not invoke any of the calls for refactoring.

are you referring to RenameHandler.invoke or something else?

Could you narrow it down to some place in your code?

E.g., Interface.method(...) is passed or outputs x1 when it should be x2

?

maybe let's wait until tomorrow. The Pros may give a simple answer.

Just checked and it is not just @, or $ in the name. If I have one or the other then everything works, the links get refactored.

[[NOTICE]] ---> NOTICE.md, refactor NOTICE.md to NOTICE@.md, links become [[NOTICE@]] which is correct

[[NOTICE@]] --> NOTICE@.md refactor to NOTICE$.md, links become [[NOTICE$]] which is correct

[[NOTICE$]] --> NOTICE$.md refactor to NOTICE@.md, links become [[NOTICE@]] which is correct

When the links don't contain spaces or @$ together then moving the file invokes handleContentChange() on my Manipulator for these PSI link elements:

public class MultiMarkdownWikiPageRefManipulator implements ElementManipulator<MultiMarkdownWikiPageRef> {
    @Override public MultiMarkdownWikiPageRef handleContentChange(@NotNull MultiMarkdownWikiPageRef element, @NotNull TextRange range, String newContent) throws IncorrectOperationException {
        if (!range.equalsToRange(0, element.getTextLength())) {
            throw new IncorrectOperationException();
        }
        return handleContentChange(element, newContent);
    }

    @Override public MultiMarkdownWikiPageRef handleContentChange(@NotNull MultiMarkdownWikiPageRef element, String newContent) throws IncorrectOperationException {
        String newName = MultiMarkdownWikiLinkImpl.makeFileName(FileUtil.getNameWithoutExtension(newContent));
        return (MultiMarkdownWikiPageRef) element.setName(newName);
    }

    @NotNull @Override public TextRange getRangeInElement(@NotNull MultiMarkdownWikiPageRef element) {
        return new TextRange(0, element.getTextLength());
    }
}
 

When renaming a file the Element's setName() method gets called by the refactoring action.

All this does not happen if the text in the element contains characters that IDEA should not be concerned with because they are part of the PsiElement's text range.

I'll wait until tomorrow. Thank you very much for trying. helped me document the issue a bit better.

did you try putting a letter between these odd characters?

e.g. [[NOTICE@]] --> NOTICE@.md refactor to NOTIC@E$.md

also e.g. [[NOTICE@]] --> NOTICE@.md refactor to NO$TIC@E.md

?

Good Question and the answer is:

If there are any characters after the @ then the links' References no longer get recognized as references for refactoring.

So it is not both, just anything after the @, or in the case of $, it can have letters following it, but if I add an @ after the $, with or without letters between $ and @, the links will not be called for rename refactoring or file move refactoring.

It really is something in the IDEA doing extra checks beyond the referring PsiElement being valid and resolving the reference to the right PsiFile.

I can't figure out where it is because there are a few run()'s and executes()'s between my code and the IDEA's.

.. and if after renaming, save and sync files,
close and reopen the project,

do the links get restored (point to files, work as expected) by any chance?

No. I tried that and I also tried invalidating the cache. Nothing works if the links have unusual characters as far as IDEA identifiers are concerned.

I even tried making the WIKI_PAGE_REF elements part of the LITERAL token set for indexing, thinking it might change somehting.

To try and fix the spaces in link text issue, I modified a copy of the Default Word Scanner not to break up CODE and LITERAL elements by words, just take the whole text range as is. Nothing changed.

I also added the WordOccurrence.Kind.FOREIGN_LANGUAGE to DefaultWordScanner for CODE elements just as it does for LITERALS, to no avail.

The usageInfo, in IDEA code which holds references to element being refactored, when it gets to do the refactoring is EMPTY if the references have space or strange characters. That is where I got stuck. I don't know enough about internals to figure out why it is empty in one case and not in the other. Same set of references should be present, all resolve to the same file.

There is a difference in the code where it does computation but it is all in run() with DumbAware checking, which is beyond my familiarity at the moment. I know what dumb aware is and my parser, annotator, editors are not dumb aware. They need the indexes.

just thought of something else:

the problem may be with DefaultWordsScanner.

what you could try:

in your implementation of FindUsagesProvider:

there is likely this line:

 
@Override
public void processWords(CharSequence fileText, Processor<WordOccurrence> processor) 


explanation:
DefaultWordsScanner.stripWords
  
uses this line:
 
while(isAsciiIdentifierPart(wordEnd) || Character.isJavaIdentifierPart(wordEnd));
 
isAsciiIdentifierPart allows '$' but not '@'

Thank you. I will try your suggestion.

I am currently doing more measured trials. I too realized, after a night's rest, that the word scanner has an effect. Yesterday, was just a mad dash of looking for solutions and I missed a few side-effects of my modifications.

I found that when I replace the word scanner to not break up words at all but use the whole element text range then TAB completions actually work as you would expect: they replace all the element's text, regardless of whether it contains spaces or other characters. I put the DefaultWordScanner back and now elements with spaces only replace to the first space break in the text.

However, I noticed that in all experimentation and trials, the spell checking no longer annotates the files.

Yesterday, I came across information that discussed adjusting the code completion or refactoring offsets right before these are applied. However, I was searching, reading and trying so many things I can't remember where I saw it nor find it now. I will, it will take time.

I will try your suggestion.

Your word scanner should provide the data that token with spaces may contain file references.
DefaultWordScanner does it only for literal word tokens.
But this one part of the story.

Other part of the story is to ensure all clients will also properly work with psi reference containing space, in particular

PsiSearchHelperImpl

AFAIK you may need custom reference searcher to handle handle the case
.

Thanks Maxim, I will do some research.

Can you please clarify this for me:

I provide the references for my PsiElements in the element's getReference(). There is always only one or no reference by these elements. If they target a file then the file is the referenced element. Otherwise, it is null because the link is not valid.

The reference's resolve() method returns an instance PsiElementResolveResult(PsiFileBase extended by my class), the file to which the link resolves to.

This is the part that I am confused about.

Is the reference resove() being used or is IDEA code doing index searching and ignoring the given references?

I was under the impression that the resolve() of the reference will create a non-implicit reference from the link to the file, without the need to 'Search for references' in the text.

By default reference resolve is used to search usages in candidate files (LowLevelSearchUtil), candidate files set is retrieved via information provided by word scanners (to avoid full scan).

Basically, if the file that contains references to elements in another file, is not open and you need to decide whether to open that file and do a full Psi parse, then you need to do a quick search so that you don't open all files in the project. That makes perfect sense.

Am I right in this understanding?

Then the lexer for Markdown can't help here because the text for the lexemes does not correspond to the file name, it needs to be converted: URLDecode(text, "UTF-8").replace(' ', '-') + ".md" before it starts to resemble the file. Additionally, the path is relative to the file where the link is contained.

Where do I find information about customizing the way references are searched in the files so that I can search for the right pattern?

Can I get information in PsiSearchHelperImpl? Or I have to look elsewhere?

> customizing the way references are searched in the files so that I can search for the right pattern

 
@Override
public void processQuery(@NotNull ReferencesSearch.SearchParameters parameters,
                         @NotNull Processor<PsiReference> consumer)

Your hunch last night was correct. Thank you.

you are welcome.

I thought this would be an easier problem :p

Implementing stuff is the easy part. The hard problem with IDEA is finding what to implement, where and when.

Most of the time is spend is trying to figure out what and where needs to be overriden, extended, registered and implemented.

Once I find the right place and thing, the rest is usually smooth sailing.

The real issue I encounter all the time, and what everyone complains about: Documentation, or lack of it.

I would say that it is not the lack of Documentation, there is plenty of that. It is a lack of a good Index and search for this documentation. If I was in charge of making IDEA easier for plugin developers I would focus on making information easier to find. In the end it is somewhere in these discussions, official documentation, etc. However, the search capability of these forums sucks, plain and simple. If I make the search too precise then it finds nothing. If I make it general, I have a sea of irrelevant posts to rumage through.

I hate using google search for IntelliJ plugin dev information because it just leads to the 'pablum' docs on the official pages.

I always wind up spending hours searching. In the end the implementation is a small fraction of the search effort.

I love the product. You would have to pry it from my cold dead hands to make me part with it, but developing plugins, especially custom language ones is more frustrating than it needs to be because of this vacuum of relevant, easily searched information.

It also makes all you guys work ten times harder answering questions that a good search engine or proper information index could have answered.

I try not to bother the experts until I am really stuck.

Thank you very much for your help.

> trying to figure out what and where needs to be overriden, extended, registered and implemented.

in addition to searching web, it really helps to open this entire project in Idea:

https://github.com/JetBrains/intellij-community

it makes it much easier to search and navigate than any web equivalent.
BTW it is not necessary to setup SDK or build it to search & navigate the project.


I agree: starting points, very basic examples are essential. Overall documentation is good, above average. A few gaps which hopefully will be filled in with time.

Forum search could be better, I agree.

wildcard (*) search,

time filter?

The idea-community is the only thing that allowed me to work on the plugin. I have it as the plugin-jdk with all the sources added so I can search break point etc. I use my compiled version, from branch 141 of the source code to run all development otherwise it is impossible to make sense of what is going on.

In many cases the code base is too huge to absorb. It is also difficult to follow because of extension points, multi-threading handling, etc. It is too easy to get lost in the forest and loose track of the trees. It goes all over the place in the debugger and static examination does not help because all the action is delegated to passed parameters. Here you really need a good documentation base for the architecture and relevant implementation, but that is easier said than done. So each plugin developer winds up making the journey practically from scratch.

I am collectiing all the relevant links and notes in a Scrivener project so that I can try to put it together into a tutorial, but with advanced features that were hard to discover, or gotcha's that I got caught on.

I just have to finish this plugin to where it does what you come to expect from an IDEA supported language, then get some sleep before I embark on a journey that has challenged ones better than me. :)

> I can try to put it together into a tutorial, but with advanced features that were hard to discover, or gotcha's that I got caught on.

> finish this plugin


Good luck with both, Vladimir!

Both tasks may need more than one person's time.

The challenge - like with most projects - is to set the scope, find those interested and keep that interest alive.

In the end I think the solution is going to be simpler than feared.

So far I only tried it in the debugger by changing the value passed to SearchRequestCollector.searchWord() from CachesBasedRefSearcher.processQuery().

If I change the word passed in so that `-` are changed to ' ' like they are for the links then the links are renamed when the file is renamed. No need to change the word scanner. It seems that word search is smart enough to find broken words if the search word has a space. :D :D

This one is going to be easy, I will make my own search extension which adds a modified file name to the word search query.

Now I will do this test for file move when a file is moved to a directory. Links with a path also don't get the reference recognition they deserve.

Must say it took a lot of single stepping and setting break points.

The search is 90% of the hunt.

Just the file name word change to replace dashes '-' to spaces ' ' is enough. The path does not affect the links, only embedded spaces because these are dashes in the file name. :D:D:D:D:D

Thank you very much for your help. It was invaluable.