The AE manual sums it up better than I could.
A matte is a layer (or any of its channels) that defines the transparent areas of that layer or another layer. White defines opaque areas, and black defines transparent areas. An alpha channel is often used as a matte, but you can use a matte other than the alpha channel if you have a channel or layer that defines the desired area of transparency better than the alpha channel does, or in cases where the source image doesn’t include an alpha channel.
A mask in After Effects is a path that is used as a parameter to modify layer attributes, effects, and properties. The most common use of a mask is the modification of an alpha channel of a layer, which determines the transparency of the layer at each pixel. Another common use of a mask is as a path along which to animate text.
So when you use Keylight, you are creating an alpha channel (not a mask) on that layer. You can also use it to create a Luma Matte (my preferred method)
which defines transparency for the original unkeyed layer below.
Footage that doesn’t have enough color or luminance information to isolate discreet areas of the image requires a user-created animated mask to generate transparency (its Alpha Channel). That’s what rotoscoping is.
Masks are created with the Marquee, Pen, and Autotrace tools.
Hope that helps.
Joey Foreman
Editor/Compositor/VFX Artist