So, I finally got both compressed and uncompressed working now.
Uncompressed video capture worked just fine as is, but with DNxHD it was an ordeal.
I simply could not view the files on Mac.
I wrote up a guide on how I solved it with ffmpeg.
My guide is available here: https://ps3computing.blogspot.ca/2012/07/definitive-guide-to-using-hyperdeck.html