19 Replies Latest reply on May 21, 2012 2:43 PM by Rallymax-forum

    CS6, format for intermediates

    cfg_2451 Level 2

      I've got some footage that needs noise reduction. Since I've already got AE, I'm going to use AE's "remove grain" effect to do the job. Unless there's a better way in the Production Premium CS6 suite?

       

      I've already found that doing this from PPro by using a "replace with AE comp" dynamic link has a huge negative impact on the performace of exporting from PPro. Same as it does in CS5. And I want to avoid this on this new project.

       

      It seems the accepted way to deal with this is to render an intermediate file from AE, and replace the source file in the PPro project with this new de-noised file. I'm OK doing this, although I've never done it before.

       

      How is this normally done from AE? What's a PPro-friendly file format for the intermediate?

       

      I'm doing this from a win7 machine, and my capture CODEC is AVCHD. To make matters worse, this is a multi-cam project in PPro. Only one camera needs the de-noise treatment. Will I have to render both cameras' source files into the same intermediate format, or can I mix and match source file format (AVCHD and XYZ) in a PPro sequence?

       

      I'm sure I'm not the first noobie to face this. Surely there's a tutorial out there somewhere? But I'm not finding the right search terms to uncover it. Somebody point me in the right direction?

        • 1. Re: CS6, format for intermediates
          Rallymax-forum Level 3

          there are many digital intermediates to choose from.

          Two free popular ones are DNxHD (Windows and Mac) and Apple ProRes (encoding is Mac only)

          A video-only free lossless codec is UTlossless. It has reasonable compression and is visually lossless.

          RAW YUV is done using Microsoft AVI Lossless but is HUGE so I would recommend you stay away from that.

           

          You do not need to render everything to an intermediate. PPro can happily work with dozens of different formats at the same time.

           

          Throw the noisy clips in to AE and de-noise them. Export them to a DI and then treat them the same as you would the other original camera footage in Premiere.

           

          Rallymax.

          1 person found this helpful
          • 2. Re: CS6, format for intermediates
            cfg_2451 Level 2

            DNxHD sounds like just the ticket. If only I could figure out how to actually use it from Adobe land. Do I have to find a plug-in (I've been looking for quite a while now and haven't found one). Neither AE or Media Encoder have this functionality that I can find. But it could be that I'm looking right at it and just don't recognize it -- the Media Encoder CS6 interface is completely different than CS5 was.

             

            I must be slow -- another nudge in the right direction please. ;-)

            • 3. Re: CS6, format for intermediates
              Jim_Simon Level 8

              A video-only free lossless codec is UT

               

              Actually they're just AVI files, (or MOV on a Mac), so audio is no problem.

              • 4. Re: CS6, format for intermediates
                Jim_Simon Level 8

                dynamic link has a huge negative impact on the performace of exporting from PPro.

                 

                How so?  The files will need to be rendered at some point, and whether you do it in AE before editing or through PP during export, the total render times should be about the same.

                • 5. Re: CS6, format for intermediates
                  cfg_2451 Level 2

                  Re: Has CS6 fixed the PPro with AE dynamic links slow to render problems?

                  Jim Simon wrote:

                  How so?  The files will need to be rendered at some point, and whether you do it in AE before editing or through PP during export, the total render times should be about the same.

                   

                  http://forums.adobe.com/message/4417103#4417103

                  • 6. Re: CS6, format for intermediates
                    Jim_Simon Level 8

                    Well, here's what I noticed in a short test using the Noise filter.

                     

                    Exporting from PP via AME took 40 seconds and pegged the CPU.  Exporting from AE to the same settings and location took 44 seconds and also pegged the CPU.

                     

                    Make of that what you will.

                    • 7. Re: CS6, format for intermediates
                      Rallymax-forum Level 3

                      It is my experience that dyn link to AE caches uncompressed rendered frames and thus is a memory hog that even with massive memory will eventually cause hard faults ie page swapping to disk.

                      If you use export to a DI there is no caching so in total time it will be quicker.

                      It also makes the user experience in ppro tolerable.

                       

                      My 2bits.

                      • 8. Re: CS6, format for intermediates
                        Rallymax-forum Level 3

                        I'd be interested in seeing your benchmark using say 5 minutes of footage - ie at least 10x RAM size of uncompressed frames.

                        • 10. Re: CS6, format for intermediates
                          Rallymax-forum Level 3

                          btw Ut is still pretty big.

                          Also DHxHD and Ut are Windows codecs (ie Video for Windows) so they are applied to AVI containers. Thus they are both found in the "Microsoft AVI" export plugin. You choose the codec there (eg Uncompressed, DNxHD, Ut). From there you can get to the Ut codec configuration box to choose whether you compress as YUV420, YUV444 or RGB.

                          I would recommend YUV444 because your source is already YUV and secondly don't take another chroma sub-sampling hit. - h.264 has already thrown away 3/4 of the chroma info (coz it's YUV420), don't do more damage. (see here.... http://en.wikipedia.org/wiki/YUV#Y.27UV420p_.28and_Y.27V12_or_YV12.29_to_RGB888_conversion)

                          • 11. Re: CS6, format for intermediates
                            Jim_Simon Level 8

                            Also DHxHD and Ut are Windows codecs (ie Video for Windows) so they are applied to AVI containers.

                             

                            DNxHD has long been available for both Windows and Mac, and with version 11, UT is also now available for Macs.

                            • 12. Re: CS6, format for intermediates
                              cfg_2451 Level 2

                              Jim Simon wrote:

                               

                              Well, here's what I noticed in a short test using the Noise filter.

                               

                              Exporting from PP via AME took 40 seconds and pegged the CPU.  Exporting from AE to the same settings and location took 44 seconds and also pegged the CPU.

                               

                              Make of that what you will.

                               

                              http://help.adobe.com/en_US/aftereffects/cs/using/WS92A6319B-B1EB-4b49-A09A-9FFE83E6CD37.h tml

                               

                              Finally found the cause, documented in a document I've read uncounted times trying to figure this out. A guy on the AE forum was kind enough to make me read it and finally understand it. I didn't get it because it was just incomprehensible to me that Adobe would intentially refuse multiprocessing. And frankly, I still have problems wrapping my head around that concept. I mean, this is 2012! This is CS6!

                               

                              The important bit for this thread is just this: "Note:  A linked After Effects composition will not support Render Multiple Frames Simultaneously multiprocessing."

                               

                              Make of that what you will.

                               

                              Given this, I don't see very many alternatives to an intermediate file, using the AE render queue (which does support multiprocessing, go figure).

                              • 13. Re: CS6, format for intermediates
                                Jim_Simon Level 8

                                I always trust what I see or experience for myself over what someone else tells me.

                                 

                                Having said that, I don't know that it needs to render "multiple frames simultaneously" in order to use multicores for the work.

                                 

                                But I do know that the linked comp pegged the CPU while exporting through AME, using all four cores in my machine, and actually rendered 4 seconds faster than going directly out of AE.

                                • 14. Re: CS6, format for intermediates
                                  cfg_2451 Level 2

                                  Jim Simon wrote:

                                   

                                  I always trust what I see or experience for myself over what someone else tells me.

                                  Finally, something we can agree on.

                                  • 15. Re: CS6, format for intermediates
                                    Rallymax-forum Level 3

                                    Jim Simon wrote:

                                     

                                    ...Having said that, I don't know that it needs to render "multiple frames simultaneously" in order to use multicores for the work...

                                    As an Exporter developer I can, with absolute certainty, report to you that Adobe can't render multiple frames in parallel.

                                    The only way to get more than one core to run simulatiously is to have one core render the next frame while the exporter uses the other cores to encode the raw frame into the output format. Having said that, during the render of that frame there may be filters/effects that are multi-core (or even many core if it's offloaded to the nVidia CUDA via the Mecury Playback Engine) but the pipeline itself can't render more than one frame at a time.

                                     

                                    'hope that clears up the capabilities for you all.

                                     

                                    Rallymax

                                    • 16. Re: CS6, format for intermediates
                                      Jim_Simon Level 8

                                      The only way to get more than one core to run simulatiously is to have one core render the next frame while the exporter uses the other cores to encode the raw frame into the output format.

                                       

                                      A plausible explanation.

                                       

                                      But however it's explained, my comp exported out of PP through AME in 4 seconds less than it did directly from AE.  This belies the idea that exporting through PP is somehow crippled in comparison to an AE direct export.  It actually came out faster through PP!

                                      • 17. Re: CS6, format for intermediates
                                        Rallymax-forum Level 3

                                        AE works in 32bit float by default. Premiere Pro works in 8bit integer.

                                        It is possible that because it went via PPro the exporter detected the source footage as 8bit not the much harder to render 32bit and thus made it faster.

                                        Do the PPro test again with "Maximum Render Quality" checked (to put it in 32bit mode) to make an Apples to Apples comparison and I'm guessing rendering from AE via PPro will be slower.

                                        • 18. Re: CS6, format for intermediates
                                          Jim_Simon Level 8

                                          GPU acceleration is on, so it's already there.

                                          • 19. Re: CS6, format for intermediates
                                            Rallymax-forum Level 3

                                            Not so.

                                            Turning MPE on doesn't make the entire pipeline 32bits. It can't - because it would then break the rule in CS5 (but no longer in CS5.5 onwards) that all filters have to support YUVA4444 8bit.

                                             

                                            Yes, all MPE accelerated plugins convert the native pipeline to 32bit float [because that's the only type of number CUDA understands] and then back to the workflow type... eg YUVA4444 8bit, afterwards but that's not the entire pipeline.

                                             

                                            If you're not in Max Render Mode then the pipeline is 8bit YUVA (which is what the Exporter gets)

                                            If you turn on Max Render Quality then the pipeline is 32bit float YUVA.

                                             

                                            Most of the time the Exporter has to do some post processing on the raw frame:

                                                 1: colorspace correction -> changing bt601 to bt709.

                                                 2: full range scaling (16-235 -> 0-255)

                                                 3: then has to do its own conversion from 32bit float to 8bit integer to be in the right format for things like MPEG2 or H.264 encode - which take YUV420 8bit as input.

                                             

                                            If 1 and 2 are in 8bit mode it is about 4 times faster to do those steps vs 32bit. I speculate this is where your 4 second saving (ie 10%) is coming from.

                                             

                                            anyhow this is now far off topic...

                                            Back on the point - There are some great free DI's out there if you want to use a DI. Or you can use Dynamic Link to AE and do it directly.

                                             

                                            Rallymax.