I have the same problem with an iMac 2013 and the GTX680 MX card.
GeForce GTX 285
Quadro FX 4800
Quadro 4000
There are only 3 video card certificaded by Adobe to used with Ray-Trace in After effects. You can try to edit the raytracer_supported_cards.txt archive into the After package installation.
Adobe says that the new update 11.0.2 support new graphics cards… I hope that little problem could be solved. Anyone knows something about it??