Adobe stopped working on ray-tracing a couple of versions ago. It never worked well to begin with. You should use Cinema 4D Lite instead, which comes bundled with AE.
About all you can do is select unsujpported GPU. It looks like you have done that. You should be seeing GPU acceleration and have access to the options. That's normal for Mac now. If you are having problems then check the CUDA driver version in your System Settings and let us know what you are seeing. Some folks have had to roll back a version but my MBPro R is working just fine.