Move directory refactoring

Hi I'm trying to test a move directory refactoring for the pants plugin (https://github.com/pantsbuild/intellij-pants-plugin).
Pants is configured through BUILD files that can live anywhere in the source tree. A BUILD file can reference other BUILD files by using relative path to the project root. What I'm trying to do is make sure that when a folder containing BUILD files gets moved, the references to those BUILD files are updated accordingly.
Currently there is a ReferenceContributor that provides PsiReference resolving to PsiFiles. This allows navigating between BUILD files.

I've setup a LightCodeInsightFixtureTestCase test but could only find a moveFile method within the fixture object, so I went to create my own moveDirectory helper, based on the one from the fixture: https://github.com/backuitist/intellij-pants-plugin/blob/feature/find_usage/tests/com/twitter/intellij/pants/refactor/BuildFileRefactorTest.java

Now it seems that I'm not using correctly the MoveFilesOrDirectoriesProcessor? I would expect isReferenceTo to be called on my PsiReference with a PsiFile implementation but it's only called with a PsiDirectory... Looks like the processor isn't looking for usages other than for the root directory. I'd expect that it searches for usages recursively through the directory children.