For a 3,1 MacPro, you should select 8 bit in the “Video Bit Depth” setting on the Project Settings / Master Project Settings / Video Monitoring area. You should also enable the “Use optimized display mode during playback”.
You didn’t list your OSX version, but if you are using anything earlier than Mountain Lion, you should go to the NVIDIA site and download the Quadro 4000 for Mac driver for your specific OSX version. This driver works for the GTX series of cards as well. Since you have a 500 series card, that should work in Lion or Mountain Lion. GTX 600 series cards will only work with Mountain Lion.
You should also go into Resolve Preferences / System Overview. That will show you what Resolve is using for the GUI and GPU cards. If it’s being used as a GPU card, it will only show the card model number. If it’s being used as a GUI card, it will say GUI after the model number. You don’t want any displays connected to the 580 card.
Dwaine Maggart
Blackmagic Design DaVinci Support