I also had the memory low problem using both Vegas Movie Studio Platinum 8 and Vegas Movie Studio HD. I was rendering a scene with a lot of short WMV clips for a 3 minute music video output as a 720p (1280 x 720) WMV file. The only images were those that were made from the video editor, so they weren’t particularly large.
8 GB of RAM and I was watching with Windows Performance Monitor, which never had less than 3000 MB FREE – not just available. I would get 90% rendered and it would fail. This is obviously a failure of Vegas software to use the memory that is available in the system. ARRGHH. I wasted a couple days being frustrated by this.
I read the advice here to reduce RAM preview settings (I reduced from 256 MB to 64 MB) and threads ( I reduced from 4 to 2). This worked. Thanks.