Problem with cached (?!) PropertyReference

I have a custom ReferenceProvider which returns PropertyReference. When I unload project A and load a different project B the property keys from project A still appear while B is open. Is this a known bug or is my code at fault?