Boiling edges is a good indicator that RotoBrush may not be the tool for the job. In effect, there is insufficient contrast between your roto object and its background.
Having said that, the Refine Hard Matte effect has a Reduce Chatter prop which helps reduce boiling artefacts.
You can combine traditional roto with Rotobrush. The workflow is to RotoBrush, followed by subtracting the boiling areas and then using traditional roto techniques for this area.
HTH
RoRK
Latest AE Workshop – MoGraph Intensity – Shapes & Text
Intensive mocha & AE Training in Singapore and Other Dangerous Locations
Imagineer Systems (mocha) Certified Instructor
& Adobe After Effects CS6 ACE/ACI