Annotator shows errors which disappear and appear intermittently

I have a custom annotator for Yaml files. If the file that is referenced by the value does not exist, show error.

I am using the same YamlFileReference and JsonFileReference that I use for navigation and the navigation works perfectly.

But when I use the below code for showing errors in the yaml file, the error is shown correctly but as soon as the text/value is fixed, a couple of issues happen intermittently

- The error wont disappear immediately

- The error will start showing at another place/value which refers to an existing file

I put the breakpoints at #1 and #2 where the resolved == null and the annotator is ready to show an error. There if I debug and evaluate

resolved = yamlFileReference.resolve();  

 it is not null. But for some reason, it is null just before the check.

What can I look into to fix it? Am I missing something that is required for implementing Annotator?

public class CustomYamlValueAnnotator implements Annotator {
  @Override
  public void annotate(@NotNull final PsiElement element, @NotNull final AnnotationHolder annotationHolder) {
    if (element.getContainingFile() == null || !(element instanceof YAMLValue)) return;

    String keyName = resolvePath(() -> ((YAMLKeyValue) element.getParent().getParent().getParent()).getName());  //returns null if NPE occurs
    boolean isParentYamlKeyValue = booleanResolver(() -> element.getParent().getParent().getParent() instanceof YAMLKeyValue);

    if (isParentYamlKeyValue) {
      PsiElement resolved;
      if ("x-combine".equals(keyName)) {
        YamlFileReference yamlFileReference = new YamlFileReference(element, element.getText());
        resolved = yamlFileReference.resolve();  
        if (resolved == null) {  //BAD - sometimes returns null
          annotationHolder.createErrorAnnotation(element, "File does not exist");  //BREAKPOINT #1

        }
      } else if ("x-schema-import".equals(keyName)) {
        JsonFileReference jsonFileReference = new JsonFileReference(element, element.getText());
        resolved = jsonFileReference.resolve();  
        if (resolved == null) { //BAD - sometimes returns null
          annotationHolder.createErrorAnnotation(element, "File does not exist");  //BREAKPOINT #2

        }
      }
    }
  }

  private boolean booleanResolver(Supplier<Boolean> supplier) {
    try {
      return supplier.get();
    } catch (NullPointerException e) {
      return false;
    }
  }
}

Thanks