you GPU is not supported for the ray-traced renderer. it is an obsolete feature anyway so save the trouble and install CC2017 and you can use the Cinema4D renderer that does the same thing only better and it works on CPU, not GPU.
You should not have to replace any Optix files and you should not have to mess with the .txt file.
For the past several versions, AE has had a checkbox to enable use of "unsupported" cards. If you undo the changes you made, uninstall AE, reinstall AE, and try using that checkbox instead, it might work.
However, like Roei said, the ray-traced renderer is considered obsolete by the AE team. It is not going to see any further development and is eventually going to go away. So, if I were you, I would put my resources towards building workflows on newer features.
Olá amigo, estou com o mesmo problema, vc conseguiu solucionar?
I resolve things by using the CPU instead the GPU, and i used BG render for get fast in the render. I made a pre-render of what I want and don't use anymore Raytrace 3D.
The new C4D render engine is cool, but there's no way to put a 3D solid with Shadow Only Option. It's was the effect I want with the 3D raytrace.