What settings do you have for your multiprocessing, CPU cores, and RAM allocations?
Check out the info here. It's written for CS5, but still applies to CC. If you're rendering with all 24 threads, you probably don't have enough RAM. Check out the "Don't starve your software of RAM" article. For HD video, you'll want at least 2GB (preferably 3GB) for each thread.
isnt it possible to clean up RAM constantly like every multi CPU 3D rendering program?
Per-pixel shader evaluations simply work completely differently. 3D programs don't explicitly "clean" anything in most cases, that stuff is inherent in how to build and optimize the underlying acceleration and evaluation structures. Still, even a 3D program will get bogged down when needing to store lots of pixel data (textures) and gobble up all your RAM. That's just what this is. And since a lot of pixel math requires lots of integer operations for building lists, color tables and calculating coordinates, there is only limited potential for optimizations without redundantly duplicating the data - just like with MP rendering. Add to that general limitations like operating systems not allowing multiple processes to the same file and the resulting wait cycles and al lof that and you will see how AE's current MP model never can go beyond certain limits even on the most beefy machine... That is to say what you are seeing - within those bounds - is normal, though of course ther specifics may vary from project to project. This has been discussed a million times, so do a little digging on the forum.
The 3D rendering process itself is different, but as you say, it also needs a lot of memory. And if the render engine would not free up whats not needed in the further progress (or mark it as rewritable), it would also slow down or stop itself. But it only needs enough memory for every single frame, regardless of how long the animation is. And all cores can work on different areas of one frame. That may not be possible or even efficiant in a compositing environment, but it seems at least not completely restricted by the OS or there are ways to get around it.
I read Adobes setup guides for MP render settings when it became available and they helped a lot. I had to asign enough memory to reduce the available 24 virtual cores to 12 real rendering cores and keep enough free. And it utilizes 100% CPU power from 12 cores in hyperthreading mode, which is quite impressive for a compositing software, so i suppose the configuration is right.
But my question is not how to optimize the initial render speed, i was wondering if there is a way to maintain it over time. Or maybe it is like you said, and thats the limit of the current MP model. I would hit this limit by exporting a 30 second DPX sequence without effects for example. So it appears to me that the memory gets loaded with rendered frames which are already written to disk.
Again, im not a programmer and its amazing what they already made possible with the current MP system. But i dont believe that a solution like removing rendered frames from memory would be beyond the possibilities of the current MP model. I dont need 40 GB of memory loaded to render frame 100 - 112 , why need it on frame 900-912?
But maybe i am wrong and thats not the problem at all. I would just like to know what exactly causes the rendering to slow down over time. Maybe there would be a workaround for now. Be it a script for example that kills and restarts MP instances or something like the hidden menu to clean the cache during the export.. :)
I had the same issues on my workstation. After a lot of fiddling (Win7 64):
I've always used quicktime as the intermediate format between editing and compositing: export QT-'same as source' from Avid, import in AE, compositing, export as QT'same as source' back to the Avid timeline. But my new system has dual six core CPU's and AE did not perform anymore. Since QT is 32-bit only, Adobe has made 'Adobe QT32 Server.exe' component to make Quicktime behave on 64-bit systems. But the more cores/threads, the slower it gets. It's a bottleneck, so to speak. I switched from quicktime to tiff-sequences as intermediate format (not as handy, of course). Now AE is remarkably fast. It flies and renders as I expect it to.
Another bottleneck is hyperthreading. This feature lets the OS believe there are more cores than physically available. But when using AE (or Lightwave or other CPU-intensive software) you're using 100% of the CPU cycles anyway. Hyperthreading makes AE use all of the virtual cores, rendering more images at once, but slower, since it's the same processor power spread thinner.
So my solution was twofold: abandoning Quicktime and disableing hyperthreading.
Since tiff-sequences are not as handy I found a great little tool to watch image sequences as one file: djv_view. Google it, it's free. I would like to see image-sequence-handling appear in Bridge, though.
We just upgraded some of our machines and have 4 of the new ones that are 16 cores each with 64GB ram. In all of my testing i determined the best thing to do was turn off Multithreading also like others said. 32 copies of after effects running at same time just doesnt seem efficient.
So i run 16 threads with 3GB ram set for each and that is the fastest rendering I was able to acheive. The 3GB settings for me was about 25% faster than the 2GB setting