Is your source recording smooth? Are there no drops in framerate in the original video where your final video speeds up?
My guess is that this is more to do with your recording software. Maybe it’s recording in a variable framerate and vegas is interpreting that as constant fps. So in moments where you lose some frames it’d speed up.