I was never too happy with the results of a. difference map.
But since DR is pretty good at tracking a Power Window, you could just stack the other clips, give each one an alpha output and use such a window for the person, with a soft edge to hide it.