If you want the mask (created using roto brush)to be white in color
(matte),simply add ‘fill effect’ and change the fill color to ‘white’.
Now go ahead and duplicate the layer and simply check on the ‘invert the Foreground/Background’ field of the the roto brush and change the ‘fill color’ to ‘black’.
This will create a matte where the area you rotoed is white in color, meaning opaque and the rest of the area is black, meaning transparent.
Sorry if this is not what you are looking for.
Pls keep me updated.