I promise that you don't have "lots of memory". You're only using a 32 bit OS, meaning you can't use more than 3 gigs of RAM. I have 16 and I still want more sometimes.
There are several potential issues you may be experiencing. Errors like this get asked about often, so try this search it's all customized and stuff!
What size is your composition, and what is in it? 3d? Shadows? Large layers? What kind of footage are you using?
If it's any kind of MPG, M2T, H.264 footage...there's your problem. From Dave LaRonde at CreativeCOW.net:
If the footage you imported into AE is any kind of the following -- footage in an HDV acquisition codec, MPEG1, MPEG2, mp4, m2t, H.261 or H.264 -- you need to convert it to a different codec.
These kinds of footage use temporal, or interframe compression. They have keyframes at regular intervals, containing complete frame information. However, the frames in between do NOT have complete information. Interframe codecs toss out duplicated information.
In order to maintain peak rendering efficiency, AE needs complete information for each and every frame. But because these kinds of footage contain only partial information, AE freaks out, resulting in a wide variety of problems.
So you might check that out.
Please be more specific! Comp settings, render settings and all that. You may simply be using settings that are incompatible with a specific QT CoDec. this is an QT error, not an AE internal one, so AE is working just fine, it just cannot compress the frame to the target format.
I'm getting a similar error from time to time, although on Mac OS X.5 (AE CS3 also).
It's says 'error compressing frame, out of memory, etc...' but if I quit AE and relaunch, it works.
I did recently install an updated vid card with advanced OpenGL, which I subsequently learned is unsupported under AE CS3, so I've disabled it's features in AE. But I wonder if this is still part of the problem.
You may want to be sure your OpenGL drivers are up to date for your specific card, or disable the OpenGL and see if you still get the same errors.
Just a thought.
On my pc, I had the same error.. I disabled audio output for the file... and it exported fine.. not sure why, but it was only for that project.
My timeline had a section with lots of effects such as motion blur. So I exported just that short area of the timeline, imported it back in, then rendered the whole thing. That was the only thing that worked for me when I got this annoying "Out of memory (-108)" error.