to fix:
place a guide ruler in ae, then hand keyframe like Blaise Douros said. but any transformation will soften the image slightly.
if the pan itself is jerky, you can use warp stabilizer on the precomped transformation or
precomp, timestretch 80%, then timewarp 80% the precomp and it will average out the pan.
next time:
either pan slowwww or shoot 60fps then speed it up in post to eliminate judder and jello.
you can also add reelmotionblur or timewarp shutter blur in post. this way, you can match blurring with pan speed to anything you want, keeping post extremely flexible.