How to get rid of remembered diagrams using IDE?

If I create a diagram from a class, this diagram appears now forever in "Recent files"-part of "Search everywhere". I can get rid of it by editing "workspace.xml" (is that really only place where diagram is stored, apart explicitly saved files?), but I haven't figured out how to get rid of these diagram hits using IDE. I really don't need them to carry around in already cluttered search results.