Yes. Following cat logic, if I fits on it, I sits on it.
If your source and your matte don’t fit, then it won’t sit.
For Set Matte to work the way you want it to, both the layer it’s applied to and the layer it’s referencing have to be the same size and the same position. This always works with text and shape layers, because they are continuously rasterized to the size of the comp. When you precomp your footage and animate inside the precomp, it also stays at the size/center of the comp.