I’ve seen this before with certain video files (and this behavior persists through multiple softwares not just AE). I have not found a workaround for this other than transcoding (which is almost always a good idea). From what I have seen pewviously it has to do with some uncommon settings in the H.264 render itself and a re-render in H.264 will usually fix it (but this will be a second lossy encode, if you are transcoding just use an intermediary codec).
Where and how was the source file generated?