What masking option could you recommend me for this effect?

[Steve Rhoden]

You can easily create that effect in Vegas, by having your clip
on two timelines (one above the other)…Apply the Defocus filter
on the bottom clip and the Circle,Centre cookie cutter filter on the
top clip…from there you simply tweak until you get what you want.

Steve Rhoden
(Cow Leader)
Film Maker
Filmex Creative Media.
1-876-832-4956

[Petr Berger]

I allready tried cookie cutter, but I cant figure out how to extend the rectangle from left to right and sometimes I might need to tilt it.