That would make sense, if ray tracing was actually on. What I'm finding is that AE CC with Mavericks on a early 2013 retina MPB, 2.8ghz, 650m, 750SSD, that AE continues to give memory warnings such as this, while ray tracing preference remains off.
In addition to the error on closing, It also gives this error on startup in the little brown boxes, then it crashes, then reopening the file, AE will launch again normally. Again, no ray tracing chosen. Understood that the MBP may not have the video card horsepower for 3d ray tracing, but, without ray tracing even enabled, these errors are baffling. I'm just guessing that mavericks memory compression and AE might need to chat a little better. it does become quite a nusciance.
Also, AE CC isn't recognizing the CUDA option for 650m as Premiere Does. Latest driver 5.5.28, near new rMBP maxed, so again, baffling as to why. Hope a patch comes soon.
The big question is: what version of After Effects CC are you running. The October 31 update is supposed to help.
The error shown in the screenshot above should be fixed by the update that we're planning to release in a couple of weeks.
I just got this same error warning after quitting CC AE (v184.108.40.206). I have been having problems all day with warp stabilizer in CC AE. I tried a number of other memory settings, but nothing fixed it. The footage will warp stabilize in CS6 AE, but no go in CC AE. I started searching for memory related threads and saw this one.