Project with MP4 HD Video renders slower with Multiprocessing ON ... normal?

[TimmyCrackCorn]

I've been working on a large project that is relatively simple ... it involves shooting some products, and then apply those videos to templates I've created to showcase the product.  Basically it's one MP4 1080p video (at 35Mb/s) with one or two color-correction effects on it, masking, and some overlayed text per comp.   Oh, and maybe the footage with masks/effects duplicated and reversed so I get a nice bounce.

THE PROBLEM:  I was getting ready to render these en masse, when I decided to do a test before enabling multiprocessing for the final renders.    I noticed that with multiprocessing ON, the renders were about 3x SLOWER!  Normally, multithreading my renders in AE results in MUCH faster renders in the past.  So I'm wondering why it is so much slower with these relatively basic comps?  Is it because of the MP4 video?

Other info:

* The video is on a relatively fast RAID0 array (Caviar Blacks), and I've never had trouble streaming multiple MP4 streams in my editing apps.

* This is in AE CS6

* This is on a I7-2600K overclocked to 4.5Ghz stable

* 64 GB of RAM

* GTX 970 4GB RAM on three displays

* AE Memory/Multiprocessing set up using 6 cores with 3GB on each core.  Disk cache is ON, with 32GB reserved on an SSD drive.

* AE is running on an (SSD for what4ever that is worth.)

Any idea why this is happening?  My only guess is, again, maybe 6 streams of compressed MP4 off a single hard drive, even a RAID is taxing AE somehow?  any insight would be appreciated.  Thanks!

[Rick Gerard]

What version of AE are you using? There is no real multi processing in AE, In some previous versions there was an option to render multiple frames simultaneously which was mistaken for MP rendering. Not all effects are compatible with the render frames simultaneously so that may be the problem. Also, how are you rendering your MP4. To get the best quality you should be using the Adobe Media Encoder and doing multi pass compression. The AME has never used MP rendering.

There are not enough details in your post to give us much to go on. I suspect that there are things different in the test renders than they were in previous projects.

[Szalam]

https://forums.adobe.com/people/Rick+Gerard  wrote

What version of AE are you using? There is no real multi processing in AE, In some previous versions there was an option to render multiple frames simultaneously which was mistaken for MP rendering. Not all effects are compatible with the render frames simultaneously so that may be the problem.

Rick, the OP is using CS6. They mentioned this in their initial post.

As far as the issue goes, what format are you rendering to?

[TimmyCrackCorn]

Szalam  wrote

Rick, the OP is using CS6. They mentioned this in their initial post.

As far as the issue goes, what format are you rendering to?

Thanks, Szalam.  I am rendering back out to MP4, same bit-rate as ingest (35Mbp/s).

[Szalam]

TimmyCrackCorn  wrote

I am rendering back out to MP4, same bit-rate as ingest (35Mbp/s).

AE has never done very well with rendering to MP4s. That is likely a large part of your bottleneck right there. There is a reason (well, probably several reasons) why Adobe has removed the MP4 rendering capability from the past several versions of After Effects. Generally speaking, you get better performance rendering to a production codec out of AE and then use the Adobe Media Encoder to make your final MP4. Bonus: AME can do two-pass encoding - that helps get you a much better-looking video at smaller file sizes - and AE can't.

Back in the day, when AE still had MP4 encoding included, I even had a stock answer I would copy and paste for issues like this explaining all of that.

And Rick must have been having a bad day. He probably meant to say that AE has never had a truly multithreaded renderer (which is true). AE's multiprocessing (not present in CC 2015+, btw) just spins up several instances of AE in the background to do the rendering. There were a multitude of bugs with it (flickering in the shading of Particular, for example), certain expressions or effects could disable it, and there were oddities like you're experiencing where it could (seemingly randomly) be slower.

I agree with Andrew that the overhead of running the multiprocessing seems to be overtaking any speed advantages - possibly due to encoding from MP4 into MP4 which AE struggled with back in the CS6 days (although, better in CS6 than CS5 which was way better than CS4!)

[TimmyCrackCorn]

Szalam  wrote

AE has never done very well with rendering to MP4s. That is likely a large part of your bottleneck right there. There is a reason (well, probably several reasons) why Adobe has removed the MP4 rendering capability from the past several versions of After Effects. Generally speaking, you get better performance rendering to a production codec out of AE and then use the Adobe Media Encoder to make your final MP4. Bonus: AME can do two-pass encoding - that helps get you a much better-looking video at smaller file sizes - and AE can't.

I have had ZERO problems in all these years using MP4 directly out of AE. Was this just a Mac issue or something? It has been a godsend for getting quick samples to clients.  Going through AME just for a quick render is neither desirable or effecient.

And again, there is absolutely no reason to remove it ... give people the option.  If they WANT to go out to AME, knock yourself out.  For those of us who just want a quick render to a universally accepted playback format, let us use it for god's sake.  It makes no sense.  Was it harming other modules of AE to just leave it in there?

NOW, that being said ... I will use AME when I need ultimate-quality and even size considerations, etc.

Back in the day, when AE still had MP4 encoding included, I even had a stock answer I would copy and paste for issues like this explaining all of that.

And Rick must have been having a bad day. He probably meant to say that AE has never had a truly multithreaded renderer (which is true). AE's multiprocessing (not present in CC 2015+, btw) just spins up several instances of AE in the background to do the rendering. There were a multitude of bugs with it (flickering in the shading of Particular, for example), certain expressions or effects could disable it, and there were oddities like you're experiencing where it could (seemingly randomly) be slower.

Yep, omni-aware of all of those ... but in cases -- or projects -- where none of that applies, it could be a real time-saver.  And if it didn't, then just turn it back off.  No biggie.  Again, Adobe throws out the baby with the bathwater for no reason. 

I agree with Andrew that the overhead of running the multiprocessing seems to be overtaking any speed advantages - possibly due to encoding from MP4 into MP4 which AE struggled with back in the CS6 days (although, better in CS6 than CS5 which was way better than CS4!)

We'll see soon enough ... easy enough to test.

[TimmyCrackCorn]

NOPE! Just did a test ... encoding into different formats with MP on and off.  Same issue ... 3-4 times slower with it ON.  ​

Encoding to MP4 is ​NOT​ the bottleneck, as I suspected.​

[Rick Gerard]

As I said before MP does not work with all effects and if you have anything that is incompatible with render multiple frames simultaneously (not really MP rendering) then the flipping back and forth when rendering is going to slow things down. IOW, what you seeing is normal and expected when you have effects in a comp that are not compatible with RMFS...

[TimmyCrackCorn]

https://forums.adobe.com/people/Rick+Gerard  wrote

As I said before MP does not work with all effects and if you have anything that is incompatible with render multiple frames simultaneously (not really MP rendering) then the flipping back and forth when rendering is going to slow things down. IOW, what you seeing is normal and expected when you have effects in a comp that are not compatible with RMFS...

Rick, please stop responding if you're not going to bother reading my posts/replies in their entirety.  This is the second time you have said something that has made no sense or had zero-bearing on the conversation as a result.  Look at post #5 where I specifically said:

"Yes, I know not all effects are compatible, and was very careful not to use effects that were incompatible with multi-processing.  The list is well known at this point."

Ironically, the same post contains my noticing your glossing over of my original post.

[Rick Gerard]

Just pointing things out that seem to be ignored by others in the thread and that make a difference to users trying to understand how AE works. I'm not glossing over your posts or not reading them. I'm so sorry that I offended you so much.

I'm not going to try and explain again how AE's RMFS or MP as most prefer to call it works.

Rendering an MP4 or decoding an MP4 to create usable pixels in AE cannot be done efficiently using RMFS because the frames have to be processed one at a time. As Mylenium pointed out this cacheing and creating temp files will quickly overwhelm any advantage gained by MP rendering.

If you, or anyone else, are rendering RMFS it is probably better to assign 4 GB/core instead of 3. In my tests I got faster renders using one less core and 4GB per core than using 3 and an additional core. Your results may vary because of the effects used in your comps.

You also said in your first post:

Oh, and maybe the footage with masks/effects duplicated and reversed so I get a nice bounce.

If you reversed the playback then you are doing temporal effects which slow down RMFS considerably.

If this were my project any source video in my compositions would be in a format that is very easy on the CPU. If I had MP4 originals and was working solely in AE I would render a DI using a suitable production format. I would also be rendering a master to a production format using the Render Cue and then compress my deliverable using the the Adobe Media encoder. In most projects, especially effects heavy projects the total time spent rendering is far less using this workflow.

Your original question, as I understood it was basically why is rendering slower with MP turned on. You are apparently using MP4 video in the timeline and may be, although you did not specifically state what format you had chosen for rendering, trying to render to MP4. I also guessed that you were doing some temporal processing with you included the word "reversed" in your description.

The practical solution is to turn of RMFS or transcode your source video into something that can be effectively processed with MP turned on and make sure that you don't have any temporal effects in your comps. There never was a mystery in my mind why your renders were slower with RMFS turned on.

[TimmyCrackCorn]

https://forums.adobe.com/people/Rick+Gerard  wrote

Just pointing things out that seem to be ignored by others in the thread and that make a difference to users trying to understand how AE works. I'm not glossing over your posts or not reading them. I'm so sorry that I offended you so much.

Rick, it's not so much that you offended ... it's just that it's extremely annoying that you keep posting things that I have already addressed, making the things you are asking or even commenting on moot.  You literally JUST DID IT AGAIN with this post (although this time, the post (or portions thereof) are not completely fruitless like before), for example:

Rendering an MP4 or decoding an MP4 to create usable pixels in AE cannot be done efficiently using RMFS because the frames have to be processed one at a time. As Mylenium pointed out this cacheing and creating temp files will quickly overwhelm any advantage gained by MP rendering.

There you go again.  I literally just posted (which you went ahead and igorned as usual) that I tried it with various formats (including Canopus HQX and other low/no compression production formats) ... same exact result.  It's the post right above the one where you replied about "Various effects not being compatible" (after I explicity mentioned I checked I wasn't using such effects!) 

RICK:  IT HAS NOTHING TO DO WITH MP4'S!!!  PLEASE FOR THE LOVE OF GOD, STOP TALKING ABOUT MP4'S!!!

If you, or anyone else, are rendering RMFS it is probably better to assign 4 GB/core instead of 3. In my tests I got faster renders using one less core and 4GB per core than using 3 and an additional core. Your results may vary because of the effects used in your comps.

Okay, now that might be useful, THANK YOU RICK.  I will try that out and see if it makes a difference. I appreciate it.

You also said in your first post:

Oh, and maybe the footage with masks/effects duplicated and reversed so I get a nice bounce.

If you reversed the playback then you are doing temporal effects which slow down RMFS considerably.

Again, thank you ... I forgot about temporal effects in this regard, but do remember reading that during my research a long time ago.  I will test this theory out by rendering something without any temporal effects and see if that might be the bottle neck.

If this were my project any source video in my compositions would be in a format that is very easy on the CPU. If I had MP4 originals and was working solely in AE I would render a DI using a suitable production format. I would also be rendering a master to a production format using the Render Cue and then compress my deliverable using the the Adobe Media encoder. In most projects, especially effects heavy projects the total time spent rendering is far less using this workflow.

Well I'm glad it's not your project then.    Seriously though, I try to avoid transcoding and re-encoding as much as possible to maintain picture quality as much as possible.  Although I guess going to HQX or the like would be lossless, so theoretically shouldn't lose anything ... but they can alter them.  But my system handles the 35Mbit/s MP4's (original recorded format) just fine.  Whether in AE or my NLE, very smooth and responsive for the most part, so why bother?

Your scenario has multiple steps and multiple transcoding going on.  Extremely ineffecient and potentially footage-altering.  Correct me if I'm wrong of course.

"There never was a mystery in my mind why your renders were slower with RMFS turned on."

Well, if the temporal tests turn out to not make a difference, then their defiinitely SHOULD BE my friend, as everything else has been de-bunked.

-Tim

[Rick Gerard]

Seriously though, I try to avoid transcoding and re-encoding as much as possible to maintain picture quality as much as possible.

If you create a visually lossless DI from a compressed file you do not loose any quality. If you choose a format that introduces color or temporal compression then you do loose quality but that isn't creating a DI (digital intermediate) it's just a poor workflow. I don't need to post my resume on the subject because there is no argument from any industry professional that a DI is often part of an efficient workflow and re-compressing footage is never a good idea.

[TimmyCrackCorn]

https://forums.adobe.com/people/Rick+Gerard  wrote

Seriously though, I try to avoid transcoding and re-encoding as much as possible to maintain picture quality as much as possible.

If you create a visually lossless DI from a compressed file you do not loose any quality. If you choose a format that introduces color or temporal compression then you do loose quality but that isn't creating a DI (digital intermediate) it's just a poor workflow. I don't need to post my resume on the subject because there is no argument from any industry professional that a DI is often part of an efficient workflow and re-compressing footage is never a good idea.

Great, then we are in total agreement here ... however, in my case I am saying that there is zero need for a DI in this case (unless you have information otherwise), and thus the workflow would be terribly ineffecient (again, in THIS CASE), when I can keep it in its original format without any sacrifices whatsoever.  Especially when I have filmed and am processing over 600 individual shots!

[TimmyCrackCorn]

https://forums.adobe.com/people/Rick+Gerard  wrote

"What version of AE are you using?"

CS6, as I mentioned in my post.  I don't think you read my entire post, Rick.

There is no real multi processing in AE, In some previous versions there was an option to render multiple frames simultaneously which was mistaken for MP rendering.

Yes, CS5.5, CS6, some versions of CC all used multi-frame rendering.  Not sure why you are saying it is not MP as well.  I can watch it use all my cores when this feature is enabled, as opposed to using only one.

"Not all effects are compatible with the render frames simultaneously so that may be the problem. Also, how are you rendering your MP4. To get the best quality you should be using the Adobe Media Encoder and doing multi pass compression. The AME has never used MP rendering. "

Yes, I know not all effects are compatible, and was very careful not to use effects that were incompatible with multi-processing.  The list is well known at this point.  I am rendering out straight from AE ... I have had ZERO problems doing this, and the FORCED use of AME to export MP4 is one of a number of reasons I unsubscribed from CC.  It is completely IDIOTIC to NOT have the OPTION.  I have yet to see a rational/logical explanation for this, and I have heard ALL of the excuses.

"There are not enough details in your post to give us much to go on. I suspect that there are things different in the test renders than they were in previous projects."

Again, I don't think you read my post, or glossed over ... I thought I provided a LOT of details.  What else do you want to know?

[Mylenium]

Not sure what you expect. Inevitably decoding a clip-based source file will require as many temp files to be produced for each individual MP instance to process the data and thus the file I/O and possible memory consumption for this temporary data will far outweigh any actual speed gain for processing, if any even comes into play. And due to how AE works, those decoded temp files require to be relatively large chunks (or even the entire source file to begin with) plus AE isn't the best program when dealing with compressed media in or out in the first place. Your workflow simply makes absolutely no sense, regardless of the points Rick raised. Simply forego that MP stuff or at best use it with two cores. Everything else isn't going to be of any use.

Mylenium

[TimmyCrackCorn]

Mylenium  wrote

"Not sure what you expect. Inevitably decoding a clip-based source file will require as many temp files to be produced for each individual MP instance to process the data and thus the file I/O and possible memory consumption for this temporary data will far outweigh any actual speed gain for processing, if any even comes into play."

​Well, obviously I expected it to be faster ... ESPECIALLY since, what you're saying (if I understand you correctly) is the bottleneck is temp files.  If that is the case, the TEMP files are being written to an exceedingly fast SSD drive.  So that should be moot.  Also, I have used MP in the past with fantastic increases in render times.  I'm wondering why this is the OPPOSITE.  Memory consumption is a non-issue ... it's not even coming anywhere near consuming half my memory.

"And due to how AE works, those decoded temp files require to be relatively large chunks (or even the entire source file to begin with) plus AE isn't the best program when dealing with compressed media in or out in the first place."

​The temp files can't be very big, as the source files are typically only 10-20 ​SECONDS​ of 1080p material.  We're talking very small files, and relatively uncompressed as far as MP4's go.  

"Your workflow simply makes absolutely no sense, regardless of the points Rick raised. Simply forego that MP stuff or at best use it with two cores. Everything else isn't going to be of any use."

Mylenium

​What about my workflow "makes absolutely no sense" ??  My worflow is pretty straightforward and typical.  Could you explain further, rather than just hurl out non-sequiturs please?  Thanks.

[Andrew Yoole · Accepted solution]

I suspect this is simply because the management of your multithreaded renders is slower than the physical render.  You have quite powerful hardware, but your project doesn't sound CPU-intensive.  So the process of rendering multiple frames and stitching them all together may be producing overhead that is slower than what a single thread render can do. Multithread renders really come into their own for CPU-intensive renders, where the CPU is working hard and long for each frame. 

And just another possibility, are you using any plugins that use GPU acceleration?  In a multithread environment with only one available GPU, each frame render must wait it's turn for GPU access, potentially slowing the render overall.

[TimmyCrackCorn]

https://forums.adobe.com/people/Andrew+Yoole  wrote

I suspect this is simply because the management of your multithreaded renders is slower than the physical render.  You have quite powerful hardware, but your project doesn't sound CPU-intensive.  So the process of rendering multiple frames and stitching them all together may be producing overhead that is slower than what a single thread render can do. Multithread renders really come into their own for CPU-intensive renders, where the CPU is working hard and long for each frame. 

And just another possibility, are you using any plugins that use GPU acceleration?  In a multithread environment with only one available GPU, each frame render must wait it's turn for GPU access, potentially slowing the render overall.

OK, thank you, Andrew!  Now this makes sense!  (And thank you for the straight-forward attitude along with the content, btw.)  I think that may very well be it.  I don't think I'm using and GPU intensive plugins (basic CC stuff is all I am using mostly), but I'll double-check that as well.  I may be using some Boris CC plugins, but not sure how heavily BCC8 relies on GPU.

Thanks again,

Tim

[Kevin-Monahan]

Hi TimmyCrackCorn,

So I'm wondering why it is so much slower with these relatively basic comps?  Is it because of the MP4 video?

Doesn't sound like you have an adequate solution yet. Or have you discovered anything new since you last posted? Let us know.

Thanks,
Kevin

[TimmyCrackCorn]

Hi Kevin:

I decided to give it to Andrew ... his answer made the most sense then, and it holds up now.  Not quite a solution, but at least I believe it explains what's going on in a nutshell.  Also I can rule out it has ANYTHING to do with rendering out to MP4 (as I mentioned).

Thanks!

-Tim

[P.M.B]

What's really strange is that the i7 2600k only supports 32gigs of RAM.   And they only fit into LGA1155 MBs which are also capped at 32 gigs of RAM.

[P.M.B]

In addition to your CPU and MB not supporting the 64 gigs of RAM you claim to have,  the 2600k only has 4 cores, not 6.   Your system specs and AE render settings do not make any sense.