as for ray-tracing, your GPU does not support it. better to not use it. make sure that when you are in a 3D scene, you are using the classic 3d renderer and not Ray-Traced renderer. you can change it in the comopsition settings->advanced.
Why are you trying to use ray-tracing in the first place? Adobe gave up on that two versions ago. Use C4D Lite to do extrusions and stuff.
Hot third-party cards don't do much at all in AE.
It sounds like you've got some traces of the CUDA driver still hanging around on your system. So, this isn't really an Adobe question, but a question for whoever knows CUDA drivers because it's an issue on your computer and not with Adobe.
That being said, if you don't have a supported GPU (and you don't), the ray-traced render is prohibitively slow. Please follow the advice you've been given to use C4D Lite instead. It comes free with AE, is more powerful than the ray-traced renderer, and will render faster on your machine. Here's a great tutorial series to get you up to speed.
Go to MAC HD>Library>Frameworks and check for CUDA.framework.
Remove it if you can find one and check again.