I am having the same issue and it is super annoying. It makes FCP load up slower than usual. I would love to find out how to end this.
I personally do not like to work with reference movies because something always seems to go missing. I helped a buddy with a project that was using them, and now that the project is done and gone, FCP is still trying to find the reference clips when it loads.