You've given us quite a bit of information about your computer, but not about AE. What version of AE is it? Are your running a Dynamic Link? Does this happen no matter what you have in your timeline or is it in a specific project? What sort of things do you have on the timeline? Is it HD or SD? 3d? Lights? Open GL settings?
Just in case you're using HDV footage, here's a bit of advice from Dave LaRonde over at CreativeCOW.net
Dave's Stock Answer #1:
If the footage you imported into AE is any kind of the following -- footage in an HDV acquisition codec, MPEG1, MPEG2, mp4, m2t, H.261 or H.264 -- you need to convert it to a different codec.
These kinds of footage use temporal, or interframe compression. They have keyframes at regular intervals, containing complete frame information. However, the frames in between do NOT have complete information. Interframe codecs toss out duplicated information.
In order to maintain peak rendering efficiency, AE needs complete information for each and every frame. But because these kinds of footage contain only partial information, AE freaks out, resulting in a wide variety of problems.
Make sure that you have the most recent version of your software. If you're using After Effects CS4, that means applying the 9.0.2 update.
Here's a blog post that makes some recommendations about memory preference settings to improve performance:
"Performance tip: Don't starve your software of RAM"
Also see the "Improve performance" page for tips.