Hi there,
There are a couple of methods for masking effects. The first is to apply the effect to an adjustment layer and mask the adjustment layer. however I don’t think that’s what you need here.
To use a Mask to contain particles is usually done within the effect itself and how you do it depends on which particle system you’re using. In Particle Playground you can use the Wall settings to use a mask to contain the particles. In Particular you can’t really use a mask in the same way but you can use a layer with an alpha channel to control the particles and create a Wall for them to bounce off in Bounce mode. Alternatively, in Air mode you can set up a spherical field to control the particles.
Let me know what software you’re using and I’ll try to give you more detailed instructions.
Hope this helps.
cheers,
Angie
Angie Taylor animation & illustration for television, film, web and devices
https://www.angietaylor.co.uk
Twitter: theangietaylor
Linkedin: theangietaylor
https://www.facebook.com/pages/Angie-Taylor/118378194869002