In AE Keylight uses Masks for holdout Instead of Bitmaps…
Use Inside & Outside mask (as garbage mattes)
make the masks on the layer,they may need to be animated.
Set them to none as a masking function
then select them in the popup in keylight.
Shake’s way is better… because you can use it for procedural mattes