This is how I would go about this:
Track the camera.
Make a solid the size and inclination of your rug.
Then project the psd-file onto the rug-solid. Do this via a light at the same position of the camera for the first frame.
As the light and the solid remain fixed, and the camera makes a 3D-move, the retouched frame should remain correct.