It looks like many people are having problems with the 5.5.28 driver on Mac OS X v10.9 (Mavericks). This is not just affecting After Effects; it is affecting all software that tries to use CUDA on this new operating system.
I suggest asking Nvidia and Apple what version of the CUDA driver they recommend for Mavericks. In the meantime, I would try rolling back to an earlier version of the CUDA driver and seeing if that works.
Before rolling back, though, you might try restarting again. I have seen several reports of multiple restarts being required for Mac OS to recognize and use the new CUDA driver.
Restarting never worked for me.
I tried rolling back to 5.5.25 and still have the same problem.
Meanwhile I am trying to download the CUDA developer package hoping it will do something.
I contacted Apple and Nvidia as well and will update you once i get an answer
Hello, I have AMD Radeon R9 M395X 4096 MB and when i open after effects 2015.3 which is the latest AE it asks me to install CUDA 5.0 or later. i downloaded and installed NVIDIA CUDA 8.0 but still cannot find raytracer in the AE Content folder. The CUDA is not installed. please help.
You do not have an NVIDIA card installed. Make sure you remove all traces of any cuda drivers or your system will be completely fouled up.
Ray-traced rendering accelerated by an appropriate NVIDIA card is no longer being developed and never worked as expected. You are spending a lot of time on a feature that is no longer and really never was a viable solution.