I asked the team and got this:
"Short answer is no, we don't participate in the Memory management preference dialog. We would need to track down where that memory is being used and whether it is something we can dynamically purge during execution. We do keep an art cache for each Ch scene, but typically these don't use that much memory. Either of his suggestions should allow him to finish his project."
"Wow, definitely never pushed the app that hard in the time or size scale. I don't even think I have a machine that I could even get close to surviving such a project. All my machines _combined_ don't have 64 GB of RAM, much less the compute horsepower it'd take to not make rendering it an exercise in pain (and possibly fire safety).
So basically it sounds like the length and size are so large that you're running into the limitations of the application. I'll DM you for some more particulars.
Hi Johannes --
I'd be curious to see a screenshot of Activity Monitor/Task Manager showing the number of Character Animator and other high memory processes and how much each one is using. Also, do you find that there are more processes than you expect based on the number of Character Animator scenes you are linking into After Effects at any given time?
This is definitely past the scale I've pushed Character Animator before, but want to make sure I understand the dimensions of how you're pushing it when trying to understand what might be accumulating so I can confirm the issue at a smaller scale.
Thanks -- Dan Tull
Thanks Dave for checking with the team. It is not the answer I was hoping for, but the answer does help me in that I now know I have to work around the issue now rather than wait.
More in response to Dan, below
If I had faster Internet I could share project files etc, but as it is, I'm literally sitting on a mountain (well, the hut is on a mountain. I'm sitting on a chair, in that hut) and data packages are brought by eagles or something. :-P
If you want, I can attach a screenshot of the task manager (I would need to "destabelize" my system first), but the screenshot might not indicate a problem. The instances of Character Animator that run as part of dynamic linking in the background seem normal. There is one instance for each CH scene used in the AE project. That makes sense.
In those two scenes in CH I have a total 3 puppets (4k puppets, between 150 and 180 mb each with cycle layers and shabang - file size tweaked to be as small as I can get them). 1 puppet hat its own scene. The other 2 share one scene.
I bring those two scenes into a 1080p AE project where a total of 5 camera angles provide medium shots of each scene and a close up of each character. All those angles make for a dialoge setup, so there are frequent cuts between the views. I set up all the other elements with dummies and color bgs instead of video footage (just dialogue audio) because editing that way is naturally much faster and means, I only need to replace the dummies and bgs before the render.
This is what my timeline looks like:
To make things fun for my system my scenes have real life 1080p footage in the background, which due to the camera setup has some nice blurry action going on. Blury-ness both affects the video and CH scenes slightly due to the nature of cameras in AE. That certainly explains why AE swallows up some processing power and Ram. But the processor has never used full power I think and memory usage on the AE part is not critical.
The length of the episodes can be as long as 40 minutes. When working on the first few (time line) minutes of the project in AE, scrubbing works and previews happen. But the more I advance in the timeline, the more RAM gets used by the 2 instances of character animator (if character animator happened to be simultaniously open as a program, that 3rd instance is idle at some low RAM usage). As I understand, this happens, because through dynamic linking all the info up to that particular point in the AE timeline needs to be stored in RAM. Early in the timeline = less RAM; later in the timeline = more RAM. When I had 32 gig of RAM I actually crashed the machine a few times. After the upgrade I barely managed to get to 27 minutes. It reaches 95-98% RAM much earlier and with a few prayers it survives and bit by bit renders and frees up things if there is no sudden demand by the OS that could lead to a max out I guess. That is why I believe that if RAM was managed through the global preferences, the system might remain stable and get through the render.
Currently I leave 80% of RAM to adobe products (which is plenty on a 64gig system). I guess I could really curb AE and force it to use the flash drive, so even more of the RAM is free for CH, but I suppose it would just give be a few more render minutes. As it is, I think the two instances of CH already would both hog as much as 45 gig.
So if RAM in Character animator will not be managed through the global adobe CC preferences even in the new release, then splitting the episodes is the only way to go for me. It is painful because it adds another render step (and I have several language versions coming up) but if this is the only way, it is the only way.
Got it, thanks for the details. That gives me enough to do some experiments and try to see what's being retained to cause that sort of memory accumulation.
Thanks Dan. Happy hunting!
Almost forgot to come back and explain what's up here.
When another app (AME, After Effects, etc) uses Character Animator to serve frames over dynamic link, it's important that when they ask for the same frame again, we produce the same output. Since Character Animator has complex animation state over time, to do that can require re-running the simulation from the beginning, so we cache the information (but not the pixels) required to regenerate those frames. For complex puppets that can get quite large. If the framerate is high or the scene is long, it adds up.
It'll take some work, but we have some ideas on how trade off random access performance with memory usage by either discarding older frames and accepting the need to regenerate or writing that data to disk instead of keeping it all in memory. Sorry that won't happen in time to save you the trouble on this project, but maybe the sequel!
Thanks for the report!
P.S. I like the style of those characters. I'm looking forward to seeing this when you finish!