Did you convert your files to ProRes using MPEG Streamclip or any other type of software?
I ran into a similar problem earlier this year. I shot in 24fps and when converting all my files I typed in 24 where it asked for what fps you shot. I noticed later my audio would drift as well here and there so I went back and redid everything but this time left the fps box blank because most programs will be smart enough to recognize what settings you shot with.
Hope that helps if you still haven’t solved the issue!