I’ve always used fraps with sony vegas but I’ve never had any problems until windows 7 and ati 5850 with fraps … I have another friend who uses fraps and sony vegas without problems!
I think that the problems arise because you have a reference GPU not overclocked.
When I bought the Gainward GTX 670’s I have not had the rendering of high quality that I did with the ATI 5850: both on Win7 Win8 that …. so beware!