I'm using AEcs3 and doing a roto. As I work, it "seems" like it's running out of RAM. At first I thought it was coz iTunes was open; closed it. Then it would happen again, I thought it was my Wacom tablet; unplug & plug, made sure it was updated to latest driver. It would still happen, so I reboot, flash the PRAM and have no programs opening and only using the mouse. When iTunes was on, the music would skip, like a CD. With nothing running except AE, then the mouse stops working and turns black, no beach ball, then I have control again. I even shutoff nucleo Pro. I don't know if this is an AE issue or an OS issue. This is the first time I've seen this. Any ideas?
Well, it needs to render the paint strokes and depending on their number, this can indeed cause massive RAM consumption, as they are vector-based. Therefore you may wanna pre-render the sections you already have and start with a "clean slate" on the rest of them. It will take some more effort to manage the various versions, but should make working easier. Also do not forget to purge the Undo list and RAM previews regularly, this may also help.