When you add keyframes to a property, After Effects will try to tween them (that is, interpolate the values in between the keyframes). That’s great if you’re animating, say, the position of an item, where you want an object to move through the values in between the keyframes you set — but it’s not so great if you’re using time-remapping to select from non-sequential poses.
The solution is to change the the keyframe interpolation to HOLD. This tells Ae to keep the value of the previous keyframe until the next keyframe occurs, rather than trying to tween smoothly from one to the next.
Select all your time-remapping keyframes, right-click any one of them, and select “Toggle hold keyframe” from the context menu.