What is the state of the Reduce Cache Size When System Is Low On Memory preference? That preference exists to work around a problem with memory management in Mac OS.
I recently selected that preference then tried working on the project only to have the same error pop up a couple minutes later...
What effects are present in your comp besides just shape layers?
Be more specific about the contents of your shape layers too. Lots of repeaters? Large shapes? Any precomps? If so, what size are they?
What are the exact specs of both machines? Version of OS?
The project contains a lot of stuff:
-Shape layers with repeaters to create animated sprites and animated backgrounds
-created wiggly text using precomped textures with fractal noise, displacement maps and posterize time effects applied. Here is the tutorial I recreated: Jittery Type After Effects Tutorial - YouTube
-imported mini AE projects of different animated objects that I used in past projects: book with pages flipping, dollar bills flipping in stack, coins stacking up, box unfolding.
-More effects used- fill, drop shadow, bunch of track mattes, turbulent displacement, hue/saturation, exposure, cc bender- on flipping pages in book,
-Lots of precomps mostly 1920 x 1080 some bigger like 1920 x 3000
Would it be possible or helpful if I just sent my project and files for you to inspect?
Processor- 3.4 GHZ Intel Core i7
Memory- 16 GB
Graphics- AMD Radeon HD 6970M 2048 MB
Software- OS X 10.9.5
MacBook Pro 17" (2012)
Processor- (don't have laptop with me so unsure)
Memory- 8 GB
Software- OS X 10.9.2
It sounds like you're using a lot of layers and effects that take up a lot of memory and both of your computers have a small amount of memory.
16 GB, as you've discovered, is not much for AE. (It's better than 8 though!)
I would suggest finding some of your big memory hogs (precomps containing large layers, for example) and prerender them.