Help getting a stub tree/index to work? Follow
I decided to add stub tree/index support to my custom language plugin to speed up some operations. The best documentation I'm able to find for this is found here:
https://confluence.jetbrains.com/display/IDEADEV/Indexing+and+PSI+Stubs+in+IntelliJ+IDEA
but the example links are mostly dead ends, and even where they take you to the sample project, it doesn't line up anymore, e.g., the sample stubElementTypeHolder doesn't seem to exist anymore.
My plugin uses a Grammar-Kit parser so I've also followed the directions from 3.5 here using the stubClass-based approach:
https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md
At this point I've implemented every step from both of these except for the stubElementTypeHolder because honestly I'm not sure how it fits in since, unlike most other extension points, that extension point doesn't comform to any interface to help guide implementation. Even without a stubElementTypeHolder, I can see my stub index being serialized in the debugger. However, I don't ever see it being deserialized once the index is built, and getStub() always returns null in my stub-based PsiElements.
Obviously I'm missing something here, but for the life of me I can't see what it might be. I've looked at other open source plugins that implement stub trees/indices and can't see what they're doing that I'm not. I imagine I'm REALLY close to having this working, but I just can't figure out that last piece of glue required to close the deal! Any guidance is GREATLY appreciated!
One more question on this...all of the documentation I've found seems to indicate that use of information from the stub and read-through to the parsed result happens automagically, but in reality it looks like you still have to implement this read-through yourself using the idiom:
if (getStub() != null)
{
return getStub().getSomething();
}
else
{
return getSomethingTheExpensiveWay();
}
That makes sense to me, but before I make things harder than they need to be, I wanted to verify that there's not some way to have both the stub element and the PSI element implement a common interface and have the runtime know whether it can get the information from the stub when available and read-through when not available.
Again, thanks much in advance for all insights and guidance here!
Please sign in to leave a comment.
Just a quick update. I did specify a stubElementTypeHolder in my plugin.xml as the elementTypeHolderClass from my BNF. That includes the appropriate stub element types because of how I've specified elementTypeFactory for my stub elements. Unfortunately that didn't fix the issue. I still don't see any attempt to deserialize my stub elements and can't really see how it might happen.
Does anyone know what triggers deserialization of stub elements? I have to assume that I have some small but critical missing piece here.
Okay, I got this working. The all-important piece that I was missing (at least as far as I can tell!) was to use my stub file to return the root stub nodes, and to use that facility to begin walking the tree. Unfortunately I'm still seeing null stubs in my stub-based PsiElements more often than I'd expect and am trying to figure out when and why that would happen, but at least now I'm able to start taking advantage of the stubs to avoid full parsing as often as possible.
Okay, so here's an example of where I'd get a PsiFile that has a null stub tree. I have a line marker provider that I use to show base/derived relationships as line markers. It works, but right now it's slower than I'd prefer because it has to parse related files (and for derived, it has to do so recursively). This is a perfect candidate for using a stub tree/index, but SlowLineMarkersPass.doCollectInformation() is getting the PsiFile from the view provider using:
PsiElement psiRoot = viewProvider.getPsi(language);
where the language is the custom language for my plugin. The returned psiRoot is one of my plugin's PsiFile subclasses as created by my ParserDefinition.createFile(FileViewProvider) implementation. So I guess in the end, I'm responsible for the returned value, but I'm not really sure if I should be creating a file with the stub populated here. This is more of that glue that I may be missing in properly implementing a stub tree/index. Any thoughts?
How exactly are you retrieving related files? The way you normally get a stub-backed PSI element is as a result of a query to a stub index. For example, the Java inherited methods line marker provider queries the class inheritor index to find the classes extending the one for which the marker is requested, and the stub index returns stub-backed elements; the files are not parsed.
Well, I have other file-based indices for other purposes. I also use those indices to find files matching other criteria and convert the VirtualFile to a PsiFile using PsiManager. Is there some better way to do that?
UPDATE: For what it's worth, you've hit upon one of the main reasons I started down the stub tree/index path...to speed up my line marker provider when it shows inheritance relationships. Let me look more closely at the Java implementation and see if I can crack the code. Any additional pointers you might be able to provide in the interim are greatly appreciated, but this should give me a solid lead!
Okay, I made decent progress on this today, but unfortunately I'm still hitting a class of issues where my stub-based elements are coming back with null stubs. In fact, they're coming back non-null from my stub index but they contain null stubs! Under what circumstance would that happen? In other words, when I call:
myStubBasedIndex.get(simpleName, project, searchScope)
and I get back a list of stub-based elements, some of which have a non-null value for myStub and some of which do not. I would assume anything coming back from this index would have a populated stub. Any idea why that wouldn't be happening?
Thanks!
UPDATE: Debugging this a bit, it looks like in StubProcessingHelperBase.processStubsInFile(), the following is sometimes returning a PsiFile with a null stub:
PsiManager.getInstance(project).findFile(file)
As a result, any contained element will also have a null stub because there's no stub tree. This seems almost completely random, though I know it isn't. Any idea why that would happen? For what it's worth, I've forced the IDE to rebuild caches/indices and still see this issue.
A null stub will be returned if IntelliJ already has a parsed PSI tree for the file. In particular, this may happen if you have some long-lived cache structures that hold references to PSI elements below file level; in that case, the PSI tree will be always held in memory.