Try this:
1. If GPU is selected for effects in the project settings, disable it
2. Try using refine soft matte effect and instead. I find it to be more suitable for edges refinements in many cases.
Note that any of these automated effects are very sensetive to noise in you footage. If your shot was not done in optimal lighting conditions, you may have to try other approach for your edges.
Roei Tzoref
After Effects Artist & Instructor
♫ Ae Blues Tutorials