Strange problem with stub indexing

Hi all,

I've just added a new stub index to my project. I've created everything according to the doc and it all seems to work ok until I try to get elements out. When I try to retrieve the elements I get back PSI elements of a completely different type. They do seem to be valid PSI elements from the project that is being indexed, but they're not the type that the index supposedly holds. I've pasted some of the relevant code snippets below:

public class ClSymbolIndex extends StringStubIndexExtension<ClSymbol> {
  public static final StubIndexKey<String, ClSymbol> KEY = StubIndexKey.createIndexKey("clojure.symbol.index");

...


public class ClSymbolElementType extends ClStubElementType<ClSymbolStub, ClSymbol> {
  public void indexStub(ClSymbolStub stub, IndexSink sink) {
    final String name = stub.getName();
    if (name != null) {
      sink.occurrence(ClSymbolIndex.KEY, name);
    }
  }

...


What I get back is a ClListImpl, a completely unrelated type to ClSymbol, but I can't see how this could happen. Does anyone have any idea what might cause this? Or if not, any suggestions on how to debug it?

8 comments

I've debugged this a little further, but I'm not much closer to figuring out what's going on. I debugged down to StubProcessingHelperBase.processStubsInFile, and when it gets the stub from the plain list in the final chunk of the method it's definitely the wrong one. I might be crazy but I have the feeling that both times I've debugged to this point the stubTreeIndex it's using is has the same value, although that might just be my imagination. However the more pressing problem seems to be that the plain stub list (and the stub tree itself) doesn't contain any stubs of my new type, which is wrong - it definitely should have some there. I've debugged my ClSymbolElementType and it seems to be behaving correctly - shouldCreateStub returns true when it should, and when it does createStub is called and creates the stub correctly. serialize is called and seems to do the right thing. At this point I'm pretty stumped, I'm not sure how to investigate this further.

Any help much appreciated.

0

Please make sure that the binary format that you use when reading and writing data is identical. A common cause of such errors is updating the stub writing code and forgetting to update the stub reading code, or vice versa.

0

Sadly, the issue is more subtle than that. Here's what's going on. In my symbol resolution I work out for each form which symbols are actually being declared and then I cache them with some metadata on the form itself using CachedValuesManager. I use the form itself as the dependency for the CachedValue. I do this since Clojure has such flexible syntax, you can't tell during parsing which forms are actually declarations so I have an extension mechanism where I can declaratively state which symbols different forms declare and what the scope of those symbols is. I create this during resolution and then I cache it because all this is moderately complex and I also don't want to have to repeat the same code in lots of different places (I can use those same cached symbols for unused symbol inspections, for example).

The change I'm making now is to index symbols instead of hard-coded forms for def/defn/etc. So I use those cached values to decide whether to index a particular symbol. In shouldCreateStub I walk the tree upwards looking for a form containing the symbol I'm checking which declares that symbol as a declaration with namespace scope - a public symbol. The problem that is causing all this is that my cached values are not correct at this stage, IntelliJ has swapped the elements at some point. I have cached on the form a declaration for a symbol that looks exactly like the one I'm checking but is actually a different element object. I can do a getParent().getText() on them both and they look identical. getContainingFile() on them returns the same file object, but they're definitely different elements. What's confusing me is that this seems to have happened without the change to the form triggering the invalidation of the cached value.

Any idea what might be going on? I assume I need to add something more to my CachedValue dependencies but I'm not sure exactly what's going on here.

0

Brief update: I just reproduced this again, and investigated a little more. Both elements not only are from the same file, they have the same parent element object. Only one of them appears in parent.getChildren(). Both elements have isValid() == true. Neither of them were created from stubs (or at least, they both have myStub == null). Both have isPhysical() == true and neither isEquivalentTo the other.

0

Ok, I have a horrible hack for this that works for the moment. Instead of checking equality for the elements, I check that their names, parents and text ranges are equal, and this works for now. I tried adding a dependency to my CachedValue that calculated a hash of the System.identityHashCodes of the children of the form but that really killed my performance - I guess this element switching happens a lot. I'd love to understand better what's going on here and to know if there's a better solution.

0

At the suggestion of Aleksandr Podkhaliusin I tried adding PsiModificationTracker.MODIFICATION_COUNT to my dependencies, but it didn't work - I still got the same problem.

0

Sadly, it seems my hack doesn't work 100% of the time - such is the nature of hacks, I guess. I'd like to fix this properly but I'm running out of ideas. Would caching smart element pointers help here? What I can't figure out is how the children of the PsiElement are switched without it triggering cache invalidation (i.e. a PSI modification event). I thought that maybe the PsiElements were being maintained when the stub tree was switched to the full PSI tree, but the parent element in this case doesn't create a stub. This happens during startup, so I assume it's some subtlety of the stub switching that is causing this. Any help greatly appreciated.

0

Did you ever figure this out?  I'm having a similar issue with indices returning incorrect objects.

0

Please sign in to leave a comment.