Rebuilding a set of file-based indices on-demand

Created June 08, 2015 20:12

My custom language plugin has a set of file-based indices including two stub indices, and the order in which they're built is important as some leverage others as they're populated.  I've expressed these dependencies in my plugin.xml and things do build properly from scratch.

There are times that I need to rebuild these entirely because there are such fundamental changes to core data structures, and not just when the index version numbers change.  If I invalidate all caches and restart things build properly, but I'm having trouble rebuilding these reliably at runtime.  I've followed the workaround from the end of this thread:

https://devnet.jetbrains.com/message/5538843

and it almost works, but for some reason things don't quite end up in the 100% right state like they do if I invalidate/restart.  Is there some way that I can simulate the behavior of invalidate/restart for just my caches at runtime?

Thanks!

2 comments

Maxim Mossienko

Created June 09, 2015 11:51

There is no order guarantee between indices processing (except thread affinity for particular content) and each index should build its value from the provided content. But it is possible to built content for many indices at once, file content provided is user data holder, one can share the data as needed for *particular* content.

By design rebuilding indices in runtime should be rarely used,e.g. we do it when indexer depends on IDE configuration (e.g. ToDoIndex) and the configuration was changed. Other cause of runtime index rebuild is error recovery.
I wonder what is your use case for runtime index rebuild.

Scott Wells

Created June 09, 2015 14:49

Yeah, I figured an explanation might be in order.  Basically my plugin generates the class library (both system classes and installed third-party library classes) dynamically at runtime by calling a series of server APIs.  Obviously until this class library is generated, references in the local classes don't resolve properly.

For performance reasons I store some information about those references, e.g., base class and implemented interface names, etc., in the file-based indices as well in user data on the respective PsiElements using CachedValuesManager.  At the time that the class library is generated, that information is incorrect and needs to be recomputed for proper linkage.

That's when I expire all caches, but as I said, they're not quite being rebuilt 100% accurately at runtime.  If I then invalidate all caches and restart, things are good once indexing completes, but since that wipes out other caches that are expensive to rebuild as well as local history, I'd really prefer to avoid that as a workaround.

Another possibility I've considered is to scan all of the files in the class library and then in the local source roots and invalidate each in turn, basically something like (pseudocode):

for (VirtualFile classLibraryFile : classLibraryFiles)
{
    FileBasedIndex.getInstance().requestReindex(classLibraryFile);
}

for (VirtualFile sourceFile : sourceFiles)
{
    FileBasedIndex.getInstance().requestReindex(sourceFile);
}

Any thoughts on whether that might be a reasonable approach?  I'm going to dive into it shortly and test the waters myself...

UPDATE: You know what?  That latest approach seems like it's going to work!  I also need to add a cache dependency (ModificationTracker) for the class library to force eviction when it's regenerated, but I think that may do the trick.  I'll report back shortly.

UPDATE 2: Okay, I think this has done it for the most part.  It's not 100% perfect, but it's VERY close.  Basically some references in current open files aren't resolving properly, but I can debug that a bit and see if it's just a cache dependency issue on my end.  I'll mark this as answered.

Please sign in to leave a comment.