There’s 3 ways I would do this.
1. motion tracking would be really fast if you set the confidence level to 90% so if he turns,it will break, and you can manually drag it in a few spots.
2. primatte keyer comes with a plugging algorithm that thresholds the size groups of the color being keyed, removing small specks.
3. use my eyeball key script in reverse to alpha out the problem.
https://technicolorsoftware.hostzi.com/