It IS something to do with the animation curves, I believe. The way to fix it is to make the first ending keyframe a Hold keyframe (command-shift-H), then add a new keyframe where you want the animation to resume – it will default to another Hold keyframe so Command-click it once to turn it back into a Linear keyframe. Make your final move at the next keyframe. This will prevent anything from drifting.
Hope this helps,
kurt murphy