I have a Mac Pro 5,1 12-Core(2 x 2.4 GHz 6-Core Intel Xeon, 12 GB RAM) with an ATI Radeon HD 5770 with 1GB that's powering two 27" LED Cinema Displays via miniport. I'm running version 2.1 ATI - 1.0.29 of OpenGL.
I use my second monitor as a full screen reference when I'm working. I can't go longer than 5-10 minutes without the screen going white. My "working" monitor with the After Effects GUI is not affected at all. If I open "Video Preview" from the preferences menu and click cancel everything goes back to normal. Generally a white screen points to OpenGL but as far as I can tell I'm up to date. I have 768MB dedicated to texture memory. My output quality is set to "faster", I'm not "mirroring on computer monitor" and I'm not scaling the output to fit the monitor.
It does this in the Classic 3D engine as well as the ray-traced. I know Adobe doesn't support ATI for GPU support in the ray-traced engine(come on Adobe, I mean seriously WTF!?!)but this shouldn't have anything to do with that I wouldn't believe...but I could certainly be wrong. It only does this when working in AE so I know the monitor is fine.
Any ideas? This is really, really frustrating.
You are simply running out of hardware resources. Running two 27" screens on a 1GB graphics card is marginal at best, insane at worst. You realyl need a better card with more VRAM. You're basically already exhausting all memory on the cards with the standard stuff, levingh barely any resources for AE's hardware acceleration stuff...