I was able to do this for both H264 and ProRes using Quicktime and custom settings from Export Media (as well as cued in Media Encoder. Give it a try let me know how it goes.
You can enable PProCC to use CUDA even if the card is not supported under Project Settings. FYI, I had no problems with my card immediately after I put in the card into my MacPro.
Managed to uninstall, and reinstall drivers. Now working.
New problem. Working on a ProRes timeline with 720p23.98 footage. Decklink not outputting on broadcast monitor. However, using a DSLR timeline, footage is outputting on monitor?? Weird.