FileBaseIndex fails to return data

已回答

创建于 2021年03月10日 16:34

I try to build a cache, based on FileBasedIndexExtension. This is the code:

public class NavFileIndex extends FileBasedIndexExtension<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>> {

    public static final ID<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>> id = ID.create("navCache");

    @Override
    public @NotNull
    ID<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>> getName() {
        return id;
    }

    @Override
    public @NotNull
    DataIndexer<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>, FileContent> getIndexer() {

        return new DataIndexer<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>, FileContent>() {

            @Override
            public Map<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>> map(@NotNull FileContent fileContent) {

                HashMap<String, Map<String, Map<String, Collection<MyItem>>>> res = new HashMap<>();

                VirtualFile virtualFile = fileContent.getFile();

                if (fileContent.getPsiFile() instanceof XmlFile) {
                    XmlFile xmlFile = (XmlFile) fileContent.getPsiFile();

                    String module = getModuleName(virtualFile);

                    XmlTag tag = xmlFile.getRootTag();
                    if (tag != null) {
                        parseTag(tag, module, null, res);
                    }
                }


                Map<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>> finalMap = new HashMap<>();
                finalMap.put(fileContent.getFile().getPath(), res);
                return finalMap;
            }
        };
    }

    private String getModuleName(VirtualFile file) {
        return "TEST"; //TODO
    }

    private String getGroup(XmlTag tag) {
        return "GRP"; //TODO
    }

    private void parseTag(XmlTag tag, String module, String group, Map<String, Map<String, Map<String, Collection<MyItem>>>> cache) {
        // Parsing Code omitted
    }

    @Override
    public @NotNull
    KeyDescriptor<String> getKeyDescriptor() {
        return EnumeratorStringDescriptor.INSTANCE;
    }

    @Override
    public @NotNull
    DataExternalizer<Map<String, Map<String, Map<String, Collection<MyItem>>>>> getValueExternalizer() {

        return new MapDataExternalizer<String, Map<String, Map<String, Collection<MyItem>>>> (
                new MapDataExternalizer<String, Map<String, Collection<MyItem>>>(
                        new MapDataExternalizer<String, Collection<MyItem>>(
                                new CollectionDataExternalizer<MyItem>(new DataExternalizer<MyItem>() {
                                    @Override
                                    public void save(@NotNull DataOutput dataOutput, MyItem item) throws IOException {
                                        dataOutput.writeUTF(item.caption);
                                        dataOutput.writeUTF(item.identifier);
                                        dataOutput.write(item.priority);
                                    }

                                    @Override
                                    public MyItem read(@NotNull DataInput dataInput) throws IOException {
                                        MyItem item = new MyItem(dataInput.readUTF(), dataInput.readUTF(), dataInput.readInt());
                                        return item;
                                    }
                                }),
                                new CollectionDataExternalizer<>(new EnumeratorStringDescriptor())),
                        new CollectionDataExternalizer<>(new EnumeratorStringDescriptor())),
                new CollectionDataExternalizer<>(new EnumeratorStringDescriptor()));
    }

    @Override
    public int getVersion() {
        return 1;
    }

    @NotNull
    @Override
    public FileBasedIndex.InputFilter getInputFilter() {
        return new FileBasedIndex.InputFilter() {
            @Override
            public boolean acceptInput(@NotNull VirtualFile virtualFile) {
                return "xml".equalsIgnoreCase(virtualFile.getExtension()) && StringUtils.containsIgnoreCase(virtualFile.getName(), "nav");
            }
        };
    }

    @Override
    public boolean dependsOnFileContent() {
        return true;
    }
}

I set a breakpoint in the getIndexer() method and it was handled correctly for all files in my project.
I try to get the contents from the cache as follows:

if (!DumbService.isDumb(project)) {
  Collection<String> res = FileBasedIndex.getInstance().getAllKeys(NavFileIndex.id, project);
  for (String key : res) {
    List<Map<String, Map<String, Map<String, Collection<MyItem>>>>> cache = FileBasedIndex.getInstance().getValues(NavFileIndex.id, key, GlobalSearchScope.allScope(project));
    for (Map<String, Map<String, Map<String, Collection<MyItem>>>> item : cache) {
     // TODO
   }
  }
}

I can read the first item in the loop. When it comes to the second, it throws a ServiceNotReadyException. If I only read one item, it starts to rebuild the index immediately after that, but I don't have changed anything in the indexed files.

What could I be doing wrong?

8 条评论

Yann Cebron

创建于 2021年03月15日 17:00

Please share full sources of your plugin.

Tom L.

创建于 2021年03月15日 18:41

Upload id: 2021_03_15_9jJevXGB8CytgmCZ (files: IndexTestProject.zip, untitled5.zip)

Just click the menu action that loads data from the index. For the first file in the loop, data is retrieved, for the second file, it fails with exception

The project "untitled5" contains 2 matching xml files for testing.

Yann Cebron

创建于 2021年03月16日 15:54

Ok, it seems these things should be researched:

(1) MyItem implementation doesn't follow required rule specified in FileBasedIndexExtension javadoc:

Note, <b>V</b>-class must have {@link Object#equals(Object)} and {@link Object#hashCode()} properly defined: value deserialized
from serialized binary data should be equal to original one.

(2) in getInputFilter(), consider using com.intellij.util.indexing.DefaultFileTypeSpecificInputFilter and add filename check in override of its acceptInput()

(3) use com.intellij.util.io.EnumeratorStringDescriptor#INSTANCE everywhere

Reference: https://plugins.jetbrains.com/docs/intellij/file-based-indexes.html#top

Tom L.

创建于 2021年03月16日 16:26

I changed everything to your recommendation with the effect, that it now builds index in an endless loop on startup of IntelliJ.

I also tried to invalidate caches and also deleted the folder "plugins-sandbox\system\index\navcache" from disc.

Here is my changed code:

public class NavFileIndex extends FileBasedIndexExtension<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>> {

    public static final ID<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>> id = ID.create("navCache");

    @Override
    public @NotNull
    ID<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>> getName() {
        return id;
    }

    @Override
    public @NotNull
    DataIndexer<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>, FileContent> getIndexer() {

        return new DataIndexer<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>, FileContent>() {

            @Override
            public Map<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>> map(@NotNull FileContent fileContent) {

                HashMap<String, Map<String, Map<String, Collection<MyItem>>>> res = new HashMap<>();

                VirtualFile virtualFile = fileContent.getFile();

                if (fileContent.getPsiFile() instanceof XmlFile) {
                    XmlFile xmlFile = (XmlFile) fileContent.getPsiFile();

                    String module = getModuleName(virtualFile);

                    XmlTag tag = xmlFile.getRootTag();
                    if (tag != null) {
                        parseTag(tag, module, null, res);
                    }
                }


                Map<String, Map<String, Map<String, Map<String, Collection<MyItem>>>>> finalMap = new HashMap<>();
                finalMap.put(fileContent.getFile().getPath(), res);
                return finalMap;
            }
        };
    }

    public Integer tryParse(String value) {
        try {
            return Integer.parseInt(value);
        } catch (NumberFormatException e) {
            return null;
        }
    }

    private String getModuleName(VirtualFile file) {
        return "TEST";
    }

    private String getGroup(XmlTag tag) {
        return "Group";
    }

    private void parseTag(XmlTag tag, String module, String group, Map<String, Map<String, Map<String, Collection<MyItem>>>> cache) {

        if (tag.getParentTag() != null) {

            String identifier = tag.getAttributeValue("identifier");
            String caption = tag.getAttributeValue("caption");
            Integer priority = tryParse(tag.getAttributeValue("priority"));
            String additionalTags = tag.getAttributeValue("additionalTags");
            String categoryTags = tag.getAttributeValue("categoryTags");

            if (group != null) {

                Map<String, Map<String, Collection<MyItem>>> byAdditionalTags;
                if (cache.containsKey(additionalTags)) {
                    byAdditionalTags = cache.get(additionalTags);
                } else {
                    byAdditionalTags = new HashMap<>();
                    cache.put(additionalTags, byAdditionalTags);
                }

                Map<String, Collection<MyItem>> byModule;
                if (byAdditionalTags.containsKey(module)) {
                    byModule = byAdditionalTags.get(module);
                } else {
                    byModule = new HashMap<>();
                    byAdditionalTags.put(module, byModule);
                }

                Collection<MyItem> byCategoryTags;
                if (byModule.containsKey(categoryTags)) {
                    byCategoryTags = byModule.get(categoryTags);
                } else {
                    byCategoryTags = new ArrayList<>();
                    byModule.put(categoryTags, byCategoryTags);
                }

                MyItem node = new MyItem(identifier, caption, priority);
                byCategoryTags.add(node);
            } else {
                group = getGroup(tag); }
        }


        for (PsiElement element : tag.getChildren()) {
            if (element instanceof XmlTag) {
                XmlTag childTag = (XmlTag) element;
                if (childTag.getName().equalsIgnoreCase("NavNode")) {
                    parseTag(childTag, module, group, cache);
                }
            }
        }
    }

    @Override
    public @NotNull
    KeyDescriptor<String> getKeyDescriptor() {
        return EnumeratorStringDescriptor.INSTANCE;
    }

    @Override
    public @NotNull
    DataExternalizer<Map<String, Map<String, Map<String, Collection<MyItem>>>>> getValueExternalizer() {

        return new MapDataExternalizer<String, Map<String, Map<String, Collection<MyItem>>>>(
                new MapDataExternalizer<String, Map<String, Collection<MyItem>>>(
                        new MapDataExternalizer<String, Collection<MyItem>>(
                                new CollectionDataExternalizer<MyItem>(new DataExternalizer<MyItem>() {
                                    @Override
                                    public void save(@NotNull DataOutput dataOutput, MyItem item) throws IOException {

                                        final boolean hasCaptionNullValue = item.caption == null;
                                        dataOutput.writeBoolean(hasCaptionNullValue);
                                        dataOutput.writeUTF(hasCaptionNullValue ? "" : item.caption);

                                        final boolean hasIdentNullValue = item.identifier == null;
                                        dataOutput.writeBoolean(hasIdentNullValue);
                                        dataOutput.writeUTF(hasIdentNullValue ? "" : item.identifier);

                                        final boolean hasPrioNullValue = item.priority == null;
                                        dataOutput.writeBoolean(hasPrioNullValue);
                                        dataOutput.writeInt(hasPrioNullValue ? 0 : item.priority);
                                    }

                                    @Override
                                    public MyItem read(@NotNull DataInput dataInput) throws IOException {

                                        final boolean hasCaptionNullValue = dataInput.readBoolean();
                                        final String captionValue = dataInput.readUTF();
                                        String caption = hasCaptionNullValue ? null : captionValue;

                                        final boolean hasIdentNullValue = dataInput.readBoolean();
                                        final String identValue = dataInput.readUTF();
                                        String ident = hasIdentNullValue ? null : identValue;

                                        final boolean hasPrioNullValue = dataInput.readBoolean();
                                        final int prioValue = dataInput.readInt();
                                        Integer prio = hasPrioNullValue ? null : prioValue;

                                        MyItem item = new MyItem(caption, ident, prio);
                                        return item;
                                    }
                                }),
                                new CollectionDataExternalizer<>(EnumeratorStringDescriptor.INSTANCE)),
                        new CollectionDataExternalizer<>(EnumeratorStringDescriptor.INSTANCE)),
                new CollectionDataExternalizer<>(EnumeratorStringDescriptor.INSTANCE));
    }

    @Override
    public int getVersion() {
        return 1;
    }

    @NotNull
    @Override
    public FileBasedIndex.InputFilter getInputFilter() {
        return new DefaultFileTypeSpecificInputFilter (XmlFileType.INSTANCE) {
            @Override
            public boolean acceptInput(@NotNull VirtualFile virtualFile) {
                return StringUtils.containsIgnoreCase(virtualFile.getName(), "nav");
            }
        };
    }

    @Override
    public boolean dependsOnFileContent() {
        return true;
    }
}

public class MyItem {

    public MyItem(String identifier, String caption, Integer priority) {
        this.identifier = identifier;
        this.caption = caption;
        this.priority = priority != null ? priority : 0;
    }

    public String identifier = "";
    public String caption = "";
    public Integer priority = 0;

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        MyItem myItem = (MyItem) o;
        return Objects.equals(identifier, myItem.identifier) && Objects.equals(caption, myItem.caption) && Objects.equals(priority, myItem.priority);
    }

    @Override
    public int hashCode() {
        return Objects.hash(identifier, caption, priority);
    }
}

Yann Cebron

创建于 2021年03月16日 16:55

I'm sorry but it's quite trouble some for me to get this running. Could you please setup a real Gradle-based project and put it somewhere on GitHub? Thanks. I also noticed your save/read is not symmetric, e.g. w/r to `null` handling.

Tom L.

创建于 2021年03月16日 18:22

I don't have any GitHub account, so I uploaded it here to the JetBrains Upload service:

Upload id: 2021_03_16_76nrPkobWGG3FaYL (file: GradleIndexTest.zip)

I'm not sure what you mean by "not symetric". I do it the same way as I saw it in the source of IJ CommunityEdition in class "LombokConfigIndex". I use the same order for reading and writing, first the bool with null-status and then the string or integer

Yann Cebron

创建于 2021年03月17日 11:54

I get this error when opening your sample test project with fresh sandbox.

2021-03-17 12:37:25,070 [ 15717] ERROR - l.indexing.impl.MapReduceIndex - null
java.io.EOFException
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:614)
at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:569)
at NavFileIndex$2.read(NavFileIndex.java:174)

I struggle to understand what the semantic structure of your index should be with current K/V signature <String, Map<String, Map<String, Map<String, Collection<MyItem>>>>>.
What kind of data do you need to store? and how do you want to access it later, what is the "key" to retrieve data? could there be multiple items with same key in one file? etc. etc.
This is absolutely critical in choosing suitable "key" and "value" for your index.

Tom L.

创建于 2021年03月17日 12:57

That is the same error that I get.

The structure represents a tree with several layers. This structure exists in multiple files. When querying the index, I combine the results into one tree and add information from the current file

These are the keys(string values) from left to right

File name
AdditionalTag
ModuleName
CategoryTag

When I read the source files, I fill the index map step by step by adding new keys or, if the same combination of "AdditionalTag" "ModuleTag" and "CategoryTag" already was found in the current file, adding the item to the collection. So it is guaranteed that the combination of these 3 strings is unique within the index of every file

Simple example:

test.xml

additionalTag: A1

ModuleName: M1

CategoryTag: C1

Item1

Item2

Module Name: M2

CategoryTag: C1

Item3

CategoryTag: C3

Item4

The same keys can be contained in multiple files but the combination of all 4 keys (including the filename) is always unique

请先登录再写评论。