Neither H.264 or AVC intra contain real whole frames. After Effects must reconstitute the frames from these files before it can work with them. This is why working with transcoded ProRes files is faster.
Premiere Pro uses the Mercury playback engine which deals these formats differently, it can use GPU acceleration, essentially transcoding the files on-the-fly. As of the current release, After Effects does not use Mercury Playback.