With newer versions of After Effects, you should not mess with the supported GPU txt file. There is a single tick box to "Use unsupported cards" in the GPU information area. If that doesn't work, neither will editing the txt file. In fact, it's more likely to work than messing with the txt file.
Anyway, it might not work at all with that newer card since the ray-traced renderer works on the Optix library and NVIDIA's been changing it, but After Effects hasn't. Here's what you can try. Rename the OptiX.1.dll from the After Effects program directory to Optix_old.1.dll . This is typically the C:\Program Files\Adobe\Adobe After Effects CC 2017\Support Files directory. Then install the latest version of OptiX from the NVIDIA website (https://developer.nvidia.com/optix ). After doing these two steps, the ray tracing functionality in After Effects should be functional. Note: After removal of the OptiX dll from the After Effects directory the CPU ray tracing support will not work.
Now, all that aside, the ray-traced renderer is considered obsolete by Adobe. The After Effects team is no longer developing it (thus why some newer cards don't work with it). Their focus for 3d is deeper integration with Cinema 4D (thus, the new Cinema 4d renderer you can pick instead of the ray-traced renderer).
Also, the new, GPU-accelerated native effects (Lumetri, Fast Gaussian Blur, etc.) use entirely different technology (and should work on your new card, no problem).
tried your method but it didnt work.. now in the gpu information the only selection is cpu and in gpu it says unsupported.. playback is really slow..
Well, if that doesn't work, delete the new stuff, rename the old file back to what it was and ignore the obsolete ray-traced renderer.
Or maybe it's going fine and you just have expectations that don't match how it works. The ray-traced renderer doesn't work in super-fast real-time or anything. How slow are we talking?
What did you end up doing? It is recommended that you use the C4D renderer for the things you used to use ray-traced 3D for. Is that what you're doing?
Let us know,