If your system is physically powering off with any software interaction, you almost certainly have a hardware issue. It could be faulty RAM or a faulty video card.
Try removing the OpenGL plugin temporarily to eliminate OpenGL as the problem.
I'd suggest re-installing After Effects (or a full OS install if you can afford the time). If the problem remains, I'd take it to a qualified service person.
Are you running these two program together?
I would think so.OK you are probably running them both as 64 bit that is great that they run 64 bit and can use more RAM and even better they actually do use more RAM.
So the solution is probably more RAM, don't run them to together or lower the expectations of Photoshop and AE.
That way they won't try to use all the RAM to do this work. It will probably do the wrk pretty fast but not as fast if you had more RAM maybe.
Also AE and PS on an iMac, what about your scratch disk. Is it a very fast external drive and how is it connected?
If you don't have a fast internal drive say with at least a 32MB buffer running at 7200rpm then there you go a real contributing ffactor.
To be honest you need both more RAM and a fast start up disk and a fast external scratch.
The new 2TB drives by Seagate run at 7200 rpm and have 64 MB cache which s as important as the rotation. You can possibly make a Raptor an external scratch as well at 300 GB they are probably not arge enough s a start up.
Those are my thoughts.
16 GB of RAM a larger faster internal drive lower the performance and memory setting as well.
Turn off OpenGL usage. Due to the high-res display in that particular iMac model, the graphics processor is operating on the edge of its performance already without any other tool burdening it further. Using OpenGL features will only push it over the edge.
Wow! Thanks guys! This was quite helpful. It would seem to me the RAM is probably the problem with Photoshop, but just realized that I was importing a CS4 After Effects project into a CS5 project using import. So, when I would just open up the CS4 project CS5 would convert it and everything was fine.
I will take that advice about RAM though. 8 gigs is probably on the edge. Would you say 16 gig would be better?