Hey there, the issue you’re having may be due to ‘auto bezier’.
When you have more than 2 keyframes for position values, After Effects likes to try and smooth out your movements so they don’t seem jerky (if you look at the dotted motion path, it’s curved instead of pointed) but I find often, the auto bezier gives undesired extra movement (especially if your working with the Z access and are changing directions drastically)
To solve this, right click the desired keyframes, select ‘keyframe interpolation’, and change the Spiral Interpolation to what you want.
OR, you can use the Pen tool on your keyframes, and modify the bezier yourself (hold and drag to stretch the curve, or alt+click to make pointed again).
– S