does the gtx 580 work on mac for ray tracer in after effects?
It should. It's one of the few cards available on Mac that were supported from the start in CS6, though of course newer versions of AE and OSX may have broken that functionality.
Since the Adobe team considers the ray-tracer obsolete, I would recommend going a different route for your 3d work. It's more powerful and it's included with AE CC and AE CC 2014: Cinema 4D! See the link I gave for more info. (Or this one: Getting Started with Cinema 4D Lite and Cineware )
See this page:
After Effects Help | Supported GPUs for ray-traced 3D renderer
Retrieving data ...