How to trigger GeneralHighlightingPass operations after file based indexing process?

File based indexes should always be up to date. If your annotating code accesses the index IntelliJ will detect that the index is out of date and re-index the file. However these indexes do not create references directly, so I assume you're somehow creating them based on the contents of the index, and I'd guess that you're caching something there which is out of date. Can you detail how your references are created?

I create refrences in following way:

public abstract class AbstractComponentReferenceProviderBase<T extends ElementLocator> extends PsiReferenceProvider {

    private final FileBasedIndex fileBasedIndex = FileBasedIndex.getInstance();
 
     public PsiReference[] getReferencesByElement(@NotNull PsiElement element, @NotNull ProcessingContext context) {
        final Module module = ModuleUtil.findModuleForPsiElement(element);
        final Project project = element.getProject();
        final String dataKey = getDataKey(element);
        if (StringUtils.isBlank(dataKey)) {
            return PsiReference.EMPTY_ARRAY;
        }
        List<T> values = fileBasedIndex.getValues(getIndexId(), dataKey, getSearchScope(module, project));
        return mapElementLocatorsToPsiReferences(values, element);
    }

     protected PsiReference[] mapElementLocatorsToPsiReferences(@NotNull List<T> values, @NotNull PsiElement referencingElement) {
        final ArrayList<PsiReference> psiReferences = new ArrayList<PsiReference>(values.size());
        boolean isSelf = false;
        for (T value : values) {
            final PsiElement target = findTargetByElementLocator(value, referencingElement);
            if (target != null) {
                isSelf = isSelf(referencingElement, target);
                if (isSelf) break;
                    //creates PsiReference using founded target and adds it to psiReferences list
                createPsiReference(referencingElement, psiReferences, value, target);
            }
        }
        if (psiReferences.isEmpty() && !isSelf) {
               //this force to indicate not resolved references
            psiReferences.add(new ComponentPsiReference(null, referencingElement, createPsiReferenceRange(referencingElement), isProduceSoftReference()));
        }
        return psiReferences.toArray(new PsiReference[psiReferences.size()]);
    }

     ...

}

findTargetByElementLocator checks is text fragment pointed by ElementLocator saved in my FileBasedIndex is valid (text range, file, pointed text fragment, offset in file) but it seems not - ElementLocator is pointing text fragment for file indexed before changes made in some plain text file, so findTargetByElementLocator returns null. From this I see that file based index for this plain text file was not valid while getReferencesByElement method was called.

I do not use any kind of caching besides FileBasedIndex ;)

Hmm, interesting - my understanding is that your call to getValues() should reindex the file if it's out of date. Can you print out the values at that point, so you can see when this code is being called and also to see if the values have been correctly updated? If getValues() doesn't return up to date values, I think that would be a reportable bug in IntelliJ.

Are you calling ReferenceProvidersRegistry#getReferencesFromProviders(element) from your PsiElement's getReferences() call per the PsiReferenceProvider javadoc? And are you sure that your element's getReferences() is being called during annotation?

I removed dataKey HPP_CHECKLIST_CONFIGURATION_CREDITING_ITEMS from my file:

2014-04-11 14:35:28,111 [ 809073]   INFO - DataIndexerWithInputFilterBase - Starting reindexing for file HPP_2_Pola.csv 
2014-04-11 14:35:28,114 [ 809076]   INFO - DataIndexerWithInputFilterBase - Reindexing for file HPP_2_Pola.csv ended 
2014-04-11 14:35:28,117 [ 809079]   INFO - ComponentReferenceProviderBase - Unresolved element: HPP_CHECKLIST_CONFIGURATION_CREDIT_PROTECTION_COUNT pl.pkobp.intellij.plugins.meta.CsvElementLocator@75751629[key=HPP_CHECKLIST_CONFIGURATION_CREDIT_PROTECTION_COUNT,textOffset=19476,textRange=(19476,19527)] 
2014-04-11 14:35:28,119 [ 809081]   INFO - ComponentReferenceProviderBase - Unresolved element: HPP_CHECKLIST_CONFIGURATION_CREDIT_PROTECTION_NUMBER pl.pkobp.intellij.plugins.meta.CsvElementLocator@5a0219fd[key=HPP_CHECKLIST_CONFIGURATION_CREDIT_PROTECTION_NUMBER,textOffset=19665,textRange=(19665,19717)] 
2014-04-11 14:35:28,156 [ 809118]   INFO - ComponentReferenceProviderBase - Unresolved element: HPP_CHECKLIST_CONFIGURATION_CREDITING_ITEMS pl.pkobp.intellij.plugins.meta.CsvElementLocator@579daae6[key=HPP_CHECKLIST_CONFIGURATION_CREDITING_ITEMS,textOffset=19157,textRange=(19157,19200)] 
2014-04-11 14:35:28,158 [ 809120]   INFO - ComponentReferenceProviderBase - Unresolved element: HPP_CHECKLIST_CONFIGURATION_CREDITING_ITEM pl.pkobp.intellij.plugins.meta.CsvElementLocator@303f0efa[key=HPP_CHECKLIST_CONFIGURATION_CREDITING_ITEM,textOffset=19299,textRange=(19299,19341)] 

but FileBasedIndex returns value for HPP_CHECKLIST_CONFIGURATION_CREDITING_ITEMS - it should be removed after

2014-04-11 14:35:28,114 [ 809076]   INFO - DataIndexerWithInputFilterBase - Reindexing for file HPP_2_Pola.csv ended 

from index storage in com.intellij.util.indexing.MapReduceIndex#updateWithMap.

All other key/values from that file after removed dataKey are invalid (wrong offsets) - values returned from FileBaseIndex are from index storage before reindexing. Maybe MemoryStorage for this index requires to refresh?

Each line "ComponentReferenceProviderBase - Unresolved element:..." indicates that dataKey has existing ElementLocator value in index storage, but it is pointing to invalid place in modified plain text file.

ElementLocator (index value) class has implemented equals and hasCode properly.

I've noticed, if I modify my indexed plain text file externally (i.e. using Notepad++), IDEA detects changes and update indices poperly (no unresolved elements besides removed one).

Hmm, I'm afraid you're reaching the limits of my knowledge about the indexing. If you're sure that the indexing is happening at the right time I would write out all the values being indexed to ensure that there's no bug in your indexing code. If you're sure that's working correctly I'd file a bug against IntelliJ, although to be honest I'd expect this code to be pretty robust. Is your code open source and available somewhere?

Okay, problem looks like this:
1. I open my indexed plain text file and json file
2. Remove some line from text file
3. Switch selected editor to json file
4. IDEA starts indexing plain text file but with wrong content (contains data which existed before edition of text file) - I obtain content for indexing via com.intellij.util.indexing.FileContent#getFile, but com.intellij.util.indexing.FileContent#getPsiFile also return invalid content
5. Built-in JS annotator higlights invalid references based on invalid index built in previous step.

But if I after step 2. switch editor to another file which don't use my FileBasedIndex for resolving references, i.e. another plain text file, and then make step 3. problems dissapears - IDEA will made second indexing, but this time with proper content, so references will be properly resolved.

It is a bug in Intellij IDEA or I do something wrong? Could someone help me with this?

Workaround for reindexing with proper file content:

public class PluginStartupActivity implements StartupActivity {
    @Override
    public void runActivity(Project project) {
        project.getMessageBus().connect().subscribe(FileEditorManagerListener.FILE_EDITOR_MANAGER, new CsvFileEditorManagerListener());
    }

}

class CsvFileEditorManagerListener implements FileEditorManagerListener {

    @Override
    public void fileOpened(FileEditorManager source, VirtualFile file) {
    }

    @Override
    public void fileClosed(FileEditorManager source, VirtualFile file) {
    }

    @Override
    public void selectionChanged(FileEditorManagerEvent event) {
        final VirtualFile oldFile = event.getOldFile();
        final FileEditor oldEditor = event.getOldEditor();
        if (oldFile != null && oldFile.getFileType() == CsvFileType.CSV && oldEditor != null && oldEditor.isModified()) {
            FileDocumentManager.getInstance().saveAllDocuments();
        }
    }
}

This is bad idea from performance view to save file (IO operation) upon selection changed (mouse moving)

You need to use FileContent.getContentAsText for indexing. VirtualFile content is different from document content you edited in editor (and only after save they are the same). Complete indexer code reference is needed for better problem diagnostcs

I change my indexing code to use com.intellij.util.indexing.FileContent#getContentAsText for receieving file content and it fix my problem. Thanks for helpful advice.

Hi Marcin,

It looks like you were implementing a CSV editor? Any chance you could share some of that code? I am doing something similar.

I have custom implementation of PSI for CSV files, but it is highly integrated with mechanisms specific for my ogranization (for internal use only), so probably, it would be easier for You to create your own implementation. You can use IntelliJ tutorial to create Your own PSI implementation for CSV files: http://www.jetbrains.org/intellij/sdk/docs/tutorials/custom_language_support_tutorial.html