The mask defines the alpha channel of the layer and it’s the first thing that AE looks at. After you told it, use this alpha channel, you applied an effect that says use this alpha channel. AE’s order of operations is Masks -> Effects -> Transformations.
If you want to use both(shift channels & mask), apply the effect to your video and precompose your layer, “shift + cntl + c”. Now mask the precomposed layer.
Johnny Cuevas, Editor
Thinkck.com
“I have not failed 700 times. I have succeeded in proving that those 700 ways will not work. When I have eliminated the ways that will not work, I will find the way that will work.”
—THOMAS EDISON on inventing the light bulb.