it might be that X looks at the project resolution and expects a matte and fill to be the same resolution when doing a Mask Filter. i know that Premiere gets weird doing this so i will place my matte and fill where i want them to be on the screen and then make a full rez still of each at my project resolution and it works.