If there is no way to pull a clean key with out losing some of the shirt, and the hole is not at the edge, you can simply add another copy of the footage layer to your comp and mask around the problem area. Feather and color correct to match and your set.
If his edges move pass the mask then you could track his movement apply the track to a solid layer draw the mask as before put this solid layer above a copy of the original footage and use a tracking matte to reveal only the problem area. (apply tracking data directly to a mask would be nice – hint, hint)
So long as the masked area does not cross his edges you need not worry about masking or painting the exact hole.
TonyTony