Here’s an idea off the top of my head:
For “continuous motion” of the foreground man, I think he needs to be in a shirt that starts with stripes and is then animated into a solid.
Shoot the actor in a shirt that is striped with your final blue but with pure green stripes (for keying purposes).
Have him step to the right.
Shoot him in the red shirt, stepping left.
The BLue/Green shirt will be the foreground man, and the red shirt will be comped behind.
Then Key the green and make it match the rest of the blue in the shirt, using a mask to wipe into the final solid blue.
The big difficulty here is getting the density of the green and blue stripe to be close. But probably easier to getting the geometry correct of stripes onto a pure green screen shirt. By putting the green stripes on the blue shirt, you can use the blue with an additional keyer to help match density.
(note: if you can’t reshoot, then take the element of the red/blue striped shirt, and see if you can key the blue into a red that matches for a full red shirt – but without planning beforehand, this may not work well.
If you could post some key frames of your footage, it would be helpful.