I've got an oddball performance issue that I'm looking for some help on. I found it in AE CS5, and it's still in AE CS6, so I thought I should pursue it here. This is on a win7 platform. For the record everything is fully up to date and patched, including the display drivers.
First, some context. Most of my work is interviews and classes, mostly two cameras. Most of the interviews are one-man-band affairs with me handling everything. On a couple of occasions I've had to film in low light conditions resulting in a noisy capture from one or both cameras. Both cameras are capturing in AVCHD, 1080p30. When we get lucky we get an interviewee who is articulate and has plenty to say, so these clips can be many minutes long. If we were talking about a seven second clip here I wouldn't be wasting your valuable time. I'm talking about as much as 25 minutes of footage.
What I've done so far is send the noisy footage from PPro to AE via dynamic link and run the "reduce grain" effect. This isn't a problem editing -- I just turn the effect off in AE so it doesn't interfere. But when it comes time to export, a few seconds of such footage is painful and a few minutes is ridiculous. I had one 3.5 minute piece take 16 hours to export, or right at 10.25 seconds for every frame.
Naturally I ran win7's Task Manager / Resource Monitor to see what was going on. What I found was that the machine wasn't limited -- I couldn't find a bottleneck. It's an i7 930 machine, so four cores as eight hyper-threads. All running at around 20-30% in this case. I've got 12 GB memory, and had as much as 6 GB free. I've got three HDDs, and none of them were busy -- the target drive getting a steady stream of frames to write, but it's average queue length was zero. The system was only writing about 2MB/second. Like I said, no bottlenecks. So what's the deal?
I've since tried to isolate the problem. Using this same footage, if I render it from AE's internal render queue, it lights up the machine -- all hyperthreads running flat out (85 - 100%), all memory used, and the target disk getting hammered. It goes from 10.25 sec/frame to 1 fps, 10x faster. But if I render the same footage through Media Encoder, I'm back to a very slow render.
It appears that when AE is called up through dynamic link, it will only run on a single hyper-thread. And when AE has to touch each and every frame (like it does using a "reduce grain" effect), it effectively makes an eight hyper-thread machine into a single hyper-thread machine. Why is this? What can I do about it?
I can't find any settings that effect this in any way. I've played with Memory and Multiprocessing Preferences to no avail (my best settings using AE's internal render queue turned out to be reserving 4 GB memory for other processes, and turning on Render Multiple Frames Simultaneously with RAM allocation / background CPU at 1.5 GB resulting in 4 hyper-threads being used for this). Are there any settings that effect this?
In summary, if I turn the reduce grain effect off in AE and export from PPro, the machine blazes away and finishes my 3.5 minute export in about 35 minutes. If I turn on the reduce grain effect, it takes more than 16 hours to do the same exact job. If I just render 3.5 minutes from the camera that needs the reduce grain effect, using ME directly (taking PPro out of the equation), it takes the same 16+ hours.
So I'm thinking this is a performance problem with AE, and the way it works with dynamic linking.
Surely I'm not the first person to see this. I'm hoping therefore that there are some good workarounds. I'd rather not have to render an intermediate file out of AE if there are any good alternatives. That's a workflow that dynamic linking was supposed to eliminate. Plus, I want to avoid re-compressing these files, and I don't have the disk space to render out a lossless I-frame intermediate file (even a moderately compressed CODEC like Cineform generates a 10x bigger file).