What specifically is your problem? Have you checked whether you actually use the latest graphics drivers?
Checking the checkbox shown on the screenshot that says "Enable untested, unsupported GPU..." will probably get it going. You may also need to see if AMD has their own drivers for it like NVIDIA provides for its cards if the default OSX don't work. This might sacrifice stability. but it may work perfectly fine.
This article is a little old but it give more info on the issue
GPU (CUDA, OpenGL) features in After Effects CS6 and After Effects CC | After Effects region of interest
James Pageau wrote:
Checking the checkbox shown on the screenshot that says "Enable untested, unsupported GPU..." will probably get it going.
I'm afraid it won't.
That checkbox is only for acceleration of the ray-traced renderer. (This is an obsolete feature, by the way.)
You can never use an AMD card to accelerate it. If you want to use it, you have to use one of a specific set of NVIDIA cards since it requires CUDA and NVIDIA's Optix Library - both of which are NVIDIA-only technology.
Again, the ray-traced renderer is not the preferred tool these days. You have a much more powerful option now.
I knew the feature was end of life. I stand corrected that the checkbox is only good for getting a untested nvidia card going.