I"m developing a simple 1920x1080 comp that has a few 3D layers
(4, all bitmaps or solids), a camera, and (so far) very few key frames. It also uses a track matte in 1 place.
When I drag an item in the window (using xyz handles) or a keyframe in the timeline, the Mac goes into a holding pattern (beach balls for about 30 seconds) while AE tries to update the frame. If I make the change numerically, the change is instant.
I'm using CS4 22.214.171.124 on OS X 10.5.7 on a brand new dual quad Mac Pro 3.2GHz with 12GB RAM and a Quadro FX 5600 video card. The system is about 2 weeks old and the AE install is brand new (2 days ago).
Turn off OpenGL. Sounds like an issue with tracking the buffers and not refreshing them properly when grabbing the layers. I would assume that this is a combination of things, so maybe try playing with your monitor resolutions and settings and see if it gets better. Short of that - since it's under warranty, swap the card or trade the machine in for a new MacPro as unleashed today.