Unfortunately. I battled this with Black Magic Support for months. In the end the only thing they could tell me that might be causing it is the memory bandwidth bottleneck in the 2008 Mac Pro Tower. I had an extremely similar setup. But with a GTX-285 in the Mac, and Quadro 4000 and Red Rocket in the Cubix. In the end I had to upgrade to a newer system with the new triple channel ram in order to get sustained playback with any reliability.
Luckily I found a job that would pay enough to cover the upgrade at the right time. I never found a way to get the 2008 3.2 GHz machine to work.
D.J.