Here's a perfect example. I've been testing out a render for a sort of template we will be using on future projects. These projects end up being about 6 minutes of final video total.
A 30-second greenscreen key with some lights/shadows, cam, 3D set, etc. I'm rendering off of an overclocked i7 980X 6-core (4.3ghz) with 24GB ram. Here are the different ways I've rendered it and the vastly different results:
MP = Multiprocessor option. The first numbers represent my render time in MM:SS.
34:40 - No MP
21:48 - MP, 0 cores for other apps, 3GB per core, 2 cores rendering
17:45 - MP, 0 cores for other apps, 2GB per core, 4 cores rendering
31:30 - MP, 0 cores for other apps, 1.5GB per core, 5 cores rendering
37:11 - MP, 0 cores for other apps, 1GB per core, 9 cores rendering
You can see how finding that sweet spot can really make a difference when you're rendering out a longer file. The two extremes are just about 1:2.
Now, is there a better way to do the above without all the trial and error?
No. That is, not until the day AE gets a new render pipeline with better adaptive threading, microbucketting and all the usual cleverness plus better temporal handling. In such a world the app would simply continually probe for available resources and fire threads and sub-threads with minimal memory footprints rather than using separate processes. And such a processing model could of course use CUDA and otehr hardware acceleration or even distributed rendering in a network/ the cloud. But alas, we're not there yet and given, how Adobe has procrastinated on this for eons and instead sold us the failed promise of OpenGL, we may have to wait another few versions...
You know, another thing that's really dumb about this whole thing is that I've noticed it doesn't really matter WHAT settings you give it. If you find the sweet spot settings, AE ignores the settings you give it anyway.
For example. I'm running a render right now. I'm on my render machine that's 32 real cores (dual AMD 6727). It couldn't use all 32 cores at 3 GB ram/core. It did its calc thing and said that using 32 cores would require using only 1.5 GB ram.
I know that for this render 1.5GB/core isn't enough ram, so I bumped it up to 2GB/core. It said it could only use 24 cores. Ok, better than nothing.
So i fire up the render and after a few minutes I runm the task manager to see how much of the system it's using, and what do you know?! I found the sweet spot and--despite it telling me I was only going to be using 24 cores--I'm using all 32 cores at a full 100%.
Those are numbers you always like to see--100%. But I only get 100% activity on ALL cores on this rig when I tell it NOT to use all its cores for rendering.
And when I tell it TO use all its cores (albeit at a lower ram/core ratio), it only used all 32 cores at 40-60%
I think the trick is to find that sweet spot. It varies from compute to computer, project to project.
Well, there are internal safeguards and metrics at work that try to be all too helpful. It's AE's eternal failing - in trying to be too smart, it actualyl looks really, really dumb and prevents you from getting things done. Well, whatever. We just have to wait and see.
You're misunderstanding the meaning of the settings.
When you tell After Effects how many CPUs to use, you're specifically telling it an upper limit of how many background processes to start (one per CPU) when you're using Render Multiple Frames Simultaneously multiprocessing. This setting, however, does not limit the number of processors that can participate in multithreading. Multithreading is independent of Render Multiple Frames Simultaneously multiprocessing.
This video demonstrates the difference:
Thanks, Todd. That clears up the misunderstanding I had. I see that using cores and multiple frames at the same time are two different things. I was lumping the two concepts together. That link was helpful.
However, I guess that the semantics are out of the way, why should we have to render out our projects with different settings (box on/off, and fine tuning the ram/core ratio) in order to know which settings happen to be best for this particular project?
It wouldn't be so bad if you only had to set it once and forget about it. But every project is different and unfortunately, you one setting that might rock in one project could be the absolute worst in another.
I posted some real results of the same render above, but with different settings. If you get the sweet-spot settings your render is literally TWICE as fast as if you pick the wrong settings.
I just did another longer test last night.
4:40 hrs. one way, and I'm going on 12 hours now with about 10% to go.
EXACT SAME PROJECT
EXACT SAME EFFECTS
EXACT SAME EXPORT SETTINGS
The ONLY thing that's different is my ram-to-core ratio.
I think that's our biggest gripe. The only way I know how to find the sweet spot is really trial-and-error.
If I can do it, why can't AE?
Why can't AE cycle through the 4 (or so) settings as it renders automatically and say:
Option 1: 10 frames per minute
Option 2: 5 frames per minute
Option 3: 15 frames per minute
Option 4: 13 frames per minute.
DING, DING! We'll go with option 3. Looks like it's the best for this render.
I know I'm not a developer and I'm sure it's a bit more complicated than that, but it seems like this is exactly what we (users) do when we render, so why couldn't AE do the same until a better solution is made?
1 person found this helpful
We also want to have After Effects take care of this for you, but it is not a trivial problem to solve. We are working on it, though.
I came here looking for a solution to this as well. I've found that this issue isn't just that you have to find the sweet spot for your machine, but you also have to find the sweet spot for the files that you're using. This is mind numbing due to Multiprocessing needing to be changed on a per project basis. I use to do that trail and error thing, but it was just such a time sink I stopped seeing a time saving benefit in Multiprocessing. I'm running a 12 core Mac with 64Gb of Ram with source stored on a Dual SAS raid 10 and AE still chokes half the time if Multiprocessing is on. This usually results in the entire system becoming unresponsive and render times taking many times the amount of a regular render. If I'm lucky, I can stop the render before it gets completely unmanagable. I've been trying to nail down what differences in projects cause this while trying to adapt the settings in AE, but it's just maddening.
TFH, I know the feeling well.... "Is it going to crash?!?" and holding onto hope. I've just gotten into the very strict habit of ALWAYS saving before a render (I know, I should anyway), but I set my render settings and then save again. Because 20% of the time the computer will not even render and I know I'll have to force close.
Oh well, hopefully CS6 brings us better tales.
I must agree with Todd, even with multi-processing off, multi-threading can and will (depending on the case) still be used when rendering a single frame.
Also to mention AE is not just an application, but the application and all the plugins that are available to it (including PPro plugins).
Imagine a plugin where an effect would use the result of the previous frame to compute the next, what do you do?!...The hardest part is to make developers to adapt to the changes and I recon Adobe team knows about it.