in hindsight it’s obvious.
You have to remember that a mask is a full image, not just the shape you’ve created. When you set an image to a Mask, Motion will look at the values and determine opacity. White is opaque, black is transparent, and gray is a value of opacity somewhere in between.
When you lower the value of opacity of the blurry circle, you moving it from white to a shade of gray–making the whatever you have applied the mask to become more and more transparent.
Instead of applying a mask to your image, what you should consider doing is creating group above your background that contains the following:
1. a black solid that is larger than your canvas
2. a white circle or other shape for your vignetted area
3. a gaussian blur layer to blur everything in this group
Then set this whole group to Multiply. Presto! your custom vignette.
I change the world.