The same problem was with my system too. Sometimes it works OK, sometimes it is like in your case. Vegas is no longer a tool we could trust to have the job done. My patience is on the edge to switch to another NLE.
This is Vegas 11 hardware acceleration problem. Disable graphics card support from Options–>Preferences–>Video–>GPU acceleration of video processing. If it does not help, in the codec settings, the “encode mode” section pick “Render using CPU only”.
Hope it helps. Let us know if that resolves your problem.