How do i get AE to use the more powerfull HD5870 instead?
Probably not at all. This is very much a known behavior and utterly normal - OpenGL being an exclusive driver model, only one ICD can take the reigns at any given time and that's what the apps use, meaning somehow the NVidia stuff is pushing itself in front of the line. you may be able to get this working by turning those features off in the NVidia cards an changing some settings in the ATI controls, but there's no guarantees. It really is outside of the control of Adobe.
Thank you, any idea where i can disable this?
Its really slowing things down, especially in PS
Turn off the global 3D acceleration option in the NVidia control panel's respective sub-section. See, if that makes any difference...
i am running OSX, i dont think there is such thing!
I have almost the exact same configuration as you, except I am using a 2009 MacPro with just one GT 120 and the new Radeon 5870. Photoshop CS5 and AfterEffects CS5 seem to be selecting the slower nVidia GT 120 by default. I just submitted a feature request here- https://www.adobe.com/cfusion/mmform/index.cfm?name=wishform detailing how the GPU selected by Photoshop CS5 is the slower one, and how there shold be a way to choose manually.
It may be useful to know that there is a plug-in available from Adobe that only works with Photoshop CS4, "AllowOldGPUs.plugin" and it would enable the program to use a different GPU. Available for download here- http://www.adobe.com/support/downloads/detail.jsp?ftpID=4051
When I launch Photoshop CS4 now it will show the Radeon 5870 as the GPU to be used for OpenGL tasks. I tried to get this plug-in to work with CS5, to no avail.
Is there some way to enable this plug-in to work with CS5?