With newer versions of After Effects, you should not mess with the supported GPU txt file. There is a single tick box to "Use unsupported cards" in the GPU information area. If that doesn't work, neither will editing the txt file. In fact, it's more likely to work than messing with the txt file.
Anyway, it might not work at all with that newer card since the ray-traced renderer works on the Optix library and NVIDIA's been changing it, but After Effects hasn't. Here's what you can try. Rename the OptiX.1.dll from the After Effects program directory to Optix_old.1.dll . This is typically the C:\Program Files\Adobe\Adobe After Effects CC 2017\Support Files directory. Then install the latest version of OptiX from the NVIDIA website (https://developer.nvidia.com/optix ). After doing these two steps, the ray tracing functionality in After Effects should be functional. Note: After removal of the OptiX dll from the After Effects directory the CPU ray tracing support will not work.
Now, all that aside, the ray-traced renderer is considered obsolete by Adobe. The After Effects team is no longer developing it (thus why some newer cards don't work with it). Their focus for 3d is deeper integration with Cinema 4D (thus, the new Cinema 4d renderer you can pick instead of the ray-traced renderer).
Also, the new, GPU-accelerated native effects (Lumetri, Fast Gaussian Blur, etc.) use entirely different technology (and should work on your new card, no problem).
tried your method but it didnt work.. now in the gpu information the only selection is cpu and in gpu it says unsupported.. playback is really slow..
Well, if that doesn't work, delete the new stuff, rename the old file back to what it was and ignore the obsolete ray-traced renderer.
Or maybe it's going fine and you just have expectations that don't match how it works. The ray-traced renderer doesn't work in super-fast real-time or anything. How slow are we talking?
What did you end up doing? It is recommended that you use the C4D renderer for the things you used to use ray-traced 3D for. Is that what you're doing?
Let us know,
If you're nervous about downloading from some random dropbox, you can get the latest Optix.DLL file from NVIDIA directly. NVIDIA® OptiX™ Ray Tracing Engine | NVIDIA Developer
Reminder, this OptiX thing is only for making the ray-traced renderer work with the 1000 series graphics cards. It will not help in any other area of After Effects.
And another reminder: the ray-traced renderer is only to create 3d geometry for quick logo extrusions, etc. - it won't help with anything else in AE. All of the other GPU functions (like acceleration of motion blur), acceleration of effects (like Fractal Noise and Glow), third-party plugins that render on the GPU (like Element), etc. have NOTHING TO DO with the ray-traced renderer or the OptiX library.
That being said, if you really need to use the ray-traced renderer to make some 3d text fly around or something...
First, rename the OptiX.1.dll from the After Effects program directory to Optix_old.1.dll . This is typically in the C:\Program Files\Adobe\Adobe After Effects CC 2018\Support Files directory. Then install the latest version of OptiX from the NVIDIA website (https://developer.nvidia.com/optix ). After doing these two steps, the ray tracing functionality in After Effects should use your GPU and, thus, be useable again.
Be aware that after these steps, the CPU ray tracing support will not work on your computer (but it's so slow as to be pointless anyway, so...).