I have a 2009 which is a 4,1 and I have flashed it to 5,1 so macOS and everything thinks it is a 2010.
I also have 16 running but I am on 10.13.6. I would go to 13 before doing anything else. Also make sure you are running the NVIDIA web driver.
What happened when you tried to install the CUDA driver? Specifics help us help you.
Mine is also not standard in that I have an PCI SSD for the OS drive and a GTX 1080Ti. Media is on a RAID 0 array, which backed up to LTO tape.
Take care,
Robert