chatter reduction should be done very carefully in Rotobrush refine edge matte or refine soft matte. I sometime have to leave it off because it introduces artifacts that are much worse than the chatter. in all the years I have used this feature I can’t say there is a preferences – simply compare the two and see which one gives you the best result over the duration of the effect.
I see there is no clear explanation in the documentation about this exact difference. it may be just a threshold thing and very related to the type of shot you are dealing with so maybe there is not clear distinction and it’s only left to experiment for yourself which one does the better job:
https://helpx.adobe.com/after-effects/using/roto-brush-refine-matte.html
https://helpx.adobe.com/after-effects/using/roto-brush-refine-edge.html#RotoBrushampRefineEdgeRefineHardMatteandRefineSoftMatteeffectreference

Roei Tzoref
2D/VFX Generalist & Instructor
♫ AeBlues Tutorials ♫
http://www.tzoref.com