Selection is the key. Once the dress is selected you can animate HSL 2 options;
1. Select the color of the dress. It must be sufficiently unique from the rest of the colors in the frames to use the Color Replace effect.
2. Select the area of the dress using an animated 16 point garbage matte. Place your clip in V1 and another instance in V2 directly above. Apply the 16 point garbage matte to the clip in V2. Toggle animation for all 16 points and then do a frame by frame adjustment of each point…after 2 or 3 days of this nonsense, apply the Color Balance effect to V2 and animate the Hue over time.
After Effects can do this easier and better with an animated mask.