On a Core II quad processor, I had the same behavior. Task Manager showed five instances of After Effects running and they were using more than a Gig of memory total. I unchecked the box for rendering multiple frames simultaneously in Preferences -> Memory & Multiprocessing and it now opens a single instance. With that box checked, I could only Ram Preview about eight seconds at full res. With it unchecked, it renders a little slower but I can Ram Preview much longer comps. Until Adobe comes up with a more efficient way to utilize multicore processors, I'd suggest that you leave that box unchecked.