What, exact, version of AE CC are you running?
Have you installed the latest drivers for your card?
What OS are you running?
What are your other system specs?
I am running a Mac:
Intel Core i7
8 GB RAM
OS X Yosemite
and I am using the current Adobe After Effects CC (2015).
NVIDIA GeForce GT 650M
According to the system requirements, that card is approved for acceleration of the ray-traced renderer: System requirements | After Effects
You have to make sure you have the current CUDA driver. Also, Macs are set up to do "intelligent" switching between the Intel graphics and your actual GPU. You need to make sure that your GPU is the only thing being used and that it doesn't switch to the Intel graphics. It's in your system settings somewhere.
What version of Yosemite? Make sure you're updated to the latest. Apple fixed some bugs that affected After Effects in their latest update.
That being said, the ray-traced renderer is considered an obsolete feature. The After Effects team is not developing it anymore. If I were you, I would put my efforts into becoming familiar with C4D Lite and the Cineware effect. It is where the AE team is putting their development time and it's more powerful than the ray-traced renderer.