I got this from Adobe when I signed up to their Expert Technical support:
“Before I get into a workflow that will work, let me step back to the core of the problem- AE 6.5 has a bug: When you have a nested 3D layer with the Continuously Rasterize switch turned on, if masks are applied in both the master and child comps, the masks will not render properly. I confirmed that this bug has been fixed in AE7, and the customer’s project works as expected.
To get round the problem the customer doesn’t need to use nested layers. What he should be using is a track matte layer with the animated masks. Specifically, here’s how to do it:.. ”