I suspect that in order to get it to look like the camera is moving when zooming in, you will need to have a background that gets enlarged when the camera moves in. Then when the background is stationary and the object shrinks, it gives a clear differentiation between the two actions.
To make it seamless, the start frame of the zoom in and the end frame will need to look the same. It is probably easiest to do sort of a double zoom (camera zoom from the current position in AND a zoom from a far-away position to the current camera position) and blend between them during the zoom — you can get away with a lot with a fast zoom — so that you end up with the exact same background on the first and last frame.
You could also increase the motion blur shutter angle (if you haven’t already) to make the motion blur even more prominent. Composition settings>Advanced.