There is no bug. Raytrace 3D is deprecated and no longer developed. Nobody is going to spend any work on supporting newer cards. Eitehr it works via nVidia's own driver magic or it doesn't. Feel free to experiment with the "untested cards" setting and whatever other tricks you find on the Internet, but otehrwise move on to the Cinema 4D renderer or whatever is your favorite 3D plug-in.
Thanks for a quick reply.
If Raytrace 3D is deprecated and no longer developed why it still available in CC 2017?
We have enough 3d engines without it.
It's carried along as "excess baggage" for people who have bene using it for the last few years and still may have supported cards. Nothing more. Even those people are strongly advised to switch to altzernate methods and it is more than foreseeable that RT 3D will be plucked from AE completely in one of the next releases.