I had this SAME frustrating issue and could not solve it after re-boot and thought it might be RAM so I purged THEN tried to freeze and nothing. FINALLY, I realized it was in the clip that I cut down in the main comp.
So to use rotobrush you have to look ONLY at that layer and so you see the entire un-edited clip. In your comp you might trim it down or edit timing or change it at all and THIS is what affects the rotobrush’s freeze. I did some heavy time re-mapping to fit my needs as well as cut the clip down (Alt+”[” or Alt+”]”). Once I realized this, I was able to kill the time re-mapping, over extend the clip in the comp window, and then render a proxy PNG sequence if needed. I didn’t need to for my purpose BUT a proxy PNG would carry the Alpha and dramatically increase your render time for the rest of the workflow.
GL. HF.