Ya, probably just lots of time remapping. What might make a project such as this easier to produce is if you applied your time remapping to a slider with values from -2 to +2(or whatever you want). This way you could just animate the slide instead of lots of painful time remapping.
Also. have you ever put a wiggle on time remapping? It makes a pretty cool effect. You could even add a separate slider for the wiggle.
Also probably doesn’t hurt if you shoot your footage at a higher frame rate.