You don’t mention the timeline length or number of clips but It sounds like you are adding 100s of key frames to the one clip, which is your complete timeline.
If so, don’t. Split the clip into individual scenes and use windows and the tracker with window shape changes in frame mode on the tracker palette.
Also, set your timeline resolution to HD. You can reset it later.