I have been having this exact same problem and it is driving me absolutely bonkers. Just like with you, it is happening in a composition that I brought in to After Effects from a Premiere sequence.
After Effects is clearly acting bizarrely in relation to these clips. Nothing I do can eliminate the problem. When I add a new adjustment layer above the affected frame(s), it changes which frame is wrong, but the problem remains.
I am just about ready to encode out the Premiere sequence and import the file into After Effects.