My copy of Vegas 11 is no longer installed, but I did open Vegas 10 and I do see that features in the mask tool have evolved and changed in later releases (including the ability to add a square/rectangle and circle/oval presets in later versions).
I can confirm that in Vegas 10, at least, it is quite difficult to create a rectangular mask.
Dell Precision T5500
2x Intel Xeon DP E5620 @ 2400MHz; Memory: 22528MB; Chipset:Intel 5520;
Video Card : NVIDIA GeForce GTX 560 Ti
HDD: Crucial_SSD1 (256GB);WD(1000GB);WD(2000GB)
OS: Windows 7 Pro SP1 (64-bit)