13 Replies Latest reply: May 18, 2013 2:11 AM by nw42 RSS

    Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?

    Jay.dub Community Member

      My question actually is, "Will it always take this long??

       

      If you haven't already checked out my system information, I am at the tail end of of a 6 year old HP 8400.  I have (finally) received the OK to acquire a new system.  While I know that this new system (http://www.pugetsystems.com/saveconfig.php?id=111597) will vastly improve "timeline" speed and After Effects previewing & rendering and should export most CODECs much faster, I'm concerned about the straight transcoding work that I do in AME.  Specifically from ProRes to anything else.

       

      The company for whom I work stages and produces lots of live corporate events.  I create content for the event and then handle whatever the client wants done with the recorded footage afterwards.  Sometimes it's the full boat and sometimes it's just for them to share internally and post on their website.  Either way, typically the first thing that they want is to just see the footage.  So, regardless of what will become of the footage later, I run it through AME and turn it into H.264 using the YouTube friendly preset.

       

      Currently, on my aging machine, AME is using only ~30% of my 8 cores.  Only 7-8 GB of my 20GB of ram and obviously 0% of my GTX285.  So am I to assume that due to CODEC limitations, AME will still take this long to transcode on a machine with twice the cores (including higher clock speeds)  and 3 times the ram?  Will it simply use even less of the system and still take 22 hours??  Will I at least see a reduction in time from the move from DDR-2 to DDR-3 and the other lower latency benifits of current hardware?

       

      I was going to include the "MediaInfo" for the source and final files along with the AME log and some other info but since I've already made this a wall of text, I hold off until someone needs it to answer my question.

       

      Thanks in advance.

       

      Jason

       

      BTW - I've been using Adobe products since the mid 90's (Premiere 4.0, After Effects "Melmet") and I've been lurking here and scraping info from all of you from at least 2 forums ago This is my first post in the "new" forum. 

        • 1. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
          Jay.dub Community Member

          So that it doesn't look like I'm simply bumping my own post , let me ask a follow up question.  

           

          This flood of ProRes footage is coming from several AJA Ki Pro Racks's that we started buying and I'm wondering if DNxHD would be the better choice since we have that option.  As a former Avid addict, recovering.  (4 years off of the wagon and now 100% Adobe) I do recall that DNxHD and ProRes were nearly identical mathematically, right?  But is DNxHD "better" at scaling with system resources?

           

          Does DNxHD make better use of processor and ram?  All things being equal, would DNxHD transcode faster to h.264?  or is all of the time lost within the h.264 encode??

           

          Thanks again,

           

          Jason

          • 2. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
            SAFEHARBOR11 Community Member

            Hi Jason,

             

            How can I put things in perspective for you? It sounds like you have no idea what to expect, which is understandable when using really old hardware. And what version of Premiere are you running?

             

            Computer technology has just made leaps and bounds the last few years, nothing short of amazing! Starting with CS5, Premiere now uses the GPU (display card) to provide hardware acceleration of editing and effects, using the Mercury Playback Engine. The MPE can run on CPU only, but it makes a world of difference when using GPU hardware as well. Of course the software and OS are 64-bit now, which really helps, especially with the large amounts of RAM allowed.

             

            Years ago, it was pretty much manadatory to get a dual-processor machine for editing. Today's Core i7 quad-core machines basically have 4 processors on one chip, so it is like having multiple processors. Then there is "multithreading", so 4 cores look like 8 cores to the software.

             

            So basically, any recent Core i7 machine will do a great job of editing and exporting HD footage from Premiere CS6. Many effects, titles, layering and scaling of video can all be done without a red bar! I stacked 15 or 20 layers of 1080i AVCHD once, making multiple PIPs all over the screen, and it still played smoothly without rendering a frame.

             

            If working primarily with Premiere and Encore, then a 4-core or 6-core i7 machine will make most editors very happy. If After Effects is a big part of your workflow, and/or 3D animation, that is where a dual-XEON machine can make a difference. Really though, working in Premiere, you'd see little difference. Realtime is realtime, an $11k machine doesn't make that any faster.

             

            On a nice i7 machine costing $3-5k, you should expect H.264 exports around realtime or better. With DV footage, going to MPEG-2 DVD format, a two-hour timeline can be exported in unnder 10 minutes. Seriously. The .wav audio from a two hour timeline? Seconds to export.

             

            Puget is a respected company with nice gear and if you feel the need for "the best of the best" and have the budget to back it up, that $11k system will certainly rock. We're also an Adobe-authorized system builder and will be happy to consult and provide pricing for comparison as well.

             

            Here's a link to Dave Helmly discussing new workstation options for CS6 - http://blog.sharbor.com/blog/2012/12/adobes-dave-helmly-discusses-thunderbolt-for-pc-and-t sunami-riptide/

             

            Thank you for your consideration

             

            Jeff Pulera
            Safe Harbor Computers

            sharbor.com

            • 3. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
              Jay.dub Community Member

              Thanks Jeff, I appreciate the response.  Perhaps I didn't pose my question the right way.

               

              Forget MPE for the moment.  I'm editing right now with a yellow bar. (GTX285) I understand the benefits of higher clock speeds and more CUDA cores.  In this respect, I do have an "idea what to expect".   This is why I'm updating (finally) my system.  To gain even more.  And yes, I also have a relatively heavy AE workload.  Hence the additional cores (and memory).

               

              However, the question I'm asking is, "is it true that":

               

              If straight transcoding from ProRes to h.264 in AME is only using 30% of my current (ancient) CPUs and 30% of my current (ancient) RAM and my RAID isn't being maxed out, it is due to the limitations and (in)efficiencies of the CODEC's involved?  And no additional resources can be brought to bare that will force a CODEC to process any faster than its inherent limitation?

               

              I'm fully aware that from the perspective of the actual work that I (personally) do, editing, animating, etc. the benefits of current hardware are overwhelming.  But from the perspective of the work that the machine will be doing without me, (straight transcoding) I cannot expect to see significant speed increases.

               

              Am I correct?

               

              Thank you,

               

              Jason

               

              Oh, and I'm running PPr - CS6 6.0.3 (001 (MC:264587)) & AME - CS6 6.0.3.1

              • 4. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
                SAFEHARBOR11 Community Member

                Hi Jason,

                 

                As I wrote earlier, expect exports to be around realtime speed. If I can get access to our demo machine this afternoon for a bit, I will try a sample render from ProRes to YouTube H.264 and get you some actual numbers.

                 

                What I gather from your post is that because the current (old) machine takes 22 hours, and is only using 30% of its potential, that a new machine will still take 20 hours. Not so. Not even close

                 

                Thanks

                 

                Jeff

                • 5. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
                  Jay.dub Community Member

                  Thanks again Jeff,

                   

                  Yes that was my question. 

                   

                  Exports from Premiere are a different story, I understand that going from timeline to export benefits greatly from better hardware.  I am referring to straight transcoding in media encoder alone.  From one CODEC to another.  (no MPE, no preview files) just workhorse file conversion inside AME outside of Premiere.

                   

                  In doing my research for this new system, I had run across a handful of threads on the forum talking about how Media Encoder is being hindered by the limitations of the CODEC’s being used and not being able to “saturate” the CPUs.

                   

                  I was starting to fear that this was what I was experiencing.  It sounds like you are saying that this is not what I am experiencing and that a new system will both “transcode” & “encode” faster.

                   

                   

                  Thank you for clarifying,

                   

                  Jason

                  • 6. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
                    SAFEHARBOR11 Community Member

                    I guess I did misunderstand, I was talking about exporting direct from Premiere via AME, I never have much call to use AME by itself for batches or anything. But no, it is not going to take hours and hours either way. Will try and test today and respond.

                     

                    Jeff

                    • 7. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
                      Harm Millaard CommunityMVP

                      Jason,

                       

                      Jeff has already given you some perspective on what you can expect from a modern machine.

                       

                      Let me add some further information:

                       

                      • A three hour single track DV timeline exported to a DV .AVI can be as fast as 22 seconds, although it normally takes around 175 to 275 seconds, depending on the disk setup.
                      • A 159 second single track AVCHD timeline, loaded with effects and exported to MPEG2-DVD with MRQ turned on, takes 23 seconds on a really fast system, and normally around 50 to 70 seconds, still better than 2 - 3 x real time.

                       

                      Finally, keep in mind that the Prores codec may not be very good at threading. I don't know this for sure, but everything related to Apple, whether it is the 32 bit QuiRcktime or Prores can show weird behaviour and inferior threading.

                       

                      For PC users there is no benefit to transcoding to Prores instead of using the native format.

                      • 8. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
                        Jay.dub Community Member

                        My apologies for not being clear here.

                         

                        I thought I had mentioned in my “bump” that we have AJA Ki Pro Racks in the field now and this footage is coming to me with the initial need to simply be converted to h.264 for the clients review and for their own use prior to any editing.  (they don't want to have to deal with a gigantic intermediate CODEC) and so I have been batch transcoding the files in AME.  So, I guess for the sake of my question, pretend that Premiere doesn’t exist.

                         

                        Not to be thick headed here but I’m not certain that we are on the same page yet.

                         

                        Forgetting premiere, for the moment.  Will Adobe Media Encoder, as a standalone batch transcoder, benefit from new hardware over my current system while converting from ProRes to h.264 when only 30% my current system is currently being used?

                         

                        Again, I apologize for my obstinance but I keep hearing answers with regards to timeline exports.  I am not asking about timeline exports.

                         

                        Thank you both very much for your time,

                         

                        Jason 

                        • 9. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
                          Jay.dub Community Member

                          Alright, I’m a moron.

                           

                          I failed to mention something that I just now discovered was pertinent information. 

                           

                          I assumed that since 100% of my system wasn’t being used, that this info didn’t figure in to the equation.


                          While I was aware that rendering at Maximum Depth, Quality and using Frame Blending increased encode time, I wasn’t aware of WHY that is.    Up until the arrival of our Ki Pros and the ProRes footage, I had just been living with the encode times and I was simply blaming everything on my old computer.   Once I was seeing 22 hour encodes, I began searching for the bottleneck and noticed that only 30% was being used and I began reading about the “diminishing returns of CODEC’s” and, subsequently, had arrived at the conclusion that sparked my question.

                           

                          In testing just now I have FINALLY discovered what you guys already knew.  Any combination of 2 of these settings seems to use 85% - 100% of my processor and increases encode time by an understandable amount and therefore, it stands to reason that AME will also use up to 100% of newer hardware (up to the limit of the CODEC).

                           

                          BUT


                          Clicking ALL THREE boxes at the same time throttles everything back to 30% and I end up with a 22 hour encode.

                           

                          I had been working  with the false understanding that “render quality” only came into account with scaling the frame (which I wasn’t doing) and frame blending only applied itself when the frame rates differed from the source (which it didn’t).  Since the way I was reading the tool tip was “with these boxes checked, IF scaling or IF different frame rates were involved, THEN render times would increase” so, I have been employing all three boxes as SOP thinking that I was maximizing quality while suffering increased encode times only when it applied.


                          I think I understand now that, while diminishing CODEC performance returns does exist, my issue was simply ignorance.


                          Thank you both for your responses and I’m sorry for having wasted everyone’s time.

                           

                          Jason

                          • 10. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
                            SAFEHARBOR11 Community Member

                            Hi Jason,

                             

                            My rule of thumb is, if I choose a PRESET, and there are boxes that are UNCHECKED, I probably do NOT need to check them - Adobe has figured out the best settings for most users already, and they are unchecked for a reason ;-)

                             

                            So anyway, I did already unbox a brand-new Ninja 2 just to prove a point before I saw your admission of guilt. I recorded exactly 60 seconds of ProRes 422, just as the KiPro would, and copied that clip into my editing PC with CS6. Dropped the 1080i ProRes 422 clip into Premiere CS6, and did File > Export > Media and chose the H.264 YouTube 1080p 29.97 preset, and began the export.

                             

                            The :60 clip took exactly :75 to complete

                             

                            I then fired up Adobe Media Encoder as a stand-alone, imported ProRes clip, and exported using same  YouTube preset, and it again took exactly :75 to export. As I said from my first post, "should be close to real time". The test system is a lowly Core i7-3770 at 3.4Ghz.

                             

                            Something else I was alluding to earlier, is that an $11k system will not necessarily be a LOT faster for certain things. One might assume that the dual-XEON system would do the same encode in half the time or less, but in reality, it might shave only 5 or 8 seconds off the encode. Don't ask me why, but that is reality. Now if you are rendering a complex scene in AE, then the big machine will very likely be quite noticeably faster. For instance, using Red Giant DeNoiser plugin, I found that a dual-XEON could render about 3 times faster than an i7 quad-core.

                             

                            In any event, a 6-year-old system is not your friend when trying to get something accomplished in the edit suite.

                             

                            Thanks

                             

                            Jeff Pulera

                            Safe Harbor

                            • 11. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
                              Jay.dub Community Member

                              The 30% processor and ram load was what was tripping me up and led to my incorrect leap of logic.  It just didn’t make sense to me and I thought that the confusion between us was exporting from the timeline as opposed to just AME.

                               

                              Again, my mistake.


                              If it weren’t for the AE work, I wouldn’t be considering dual Xeons.  But most of the animation work I do is  for ultra-widescreen blended projection (9600 X 1080) typically with lots of nested comps and layer effects so, unless I’m again mistaken, I’m looking at the need of as may processors  as I can get and enough ram to satisfy each.  Or am I missing the mark there as well?

                              • 12. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
                                SAFEHARBOR11 Community Member

                                For heavy comp work and animation/3D, then a bigger workstation can be very worthwhile. If someone is primarily working in Premiere, then a 4 or 6-core is more than adequate, as Premiere can only use so much oomph behind it.

                                 

                                Jeff

                                • 13. Re: Does it really take 22 hours for AME to transcode 1 hour of ProRes apcn to the YouTube preset?
                                  nw42 Community Member

                                  Hi,

                                   

                                  this discussion is missing some important things... a fast and poweful workstation can't help with software bottlenecks in PP and Quicktime...

                                   

                                  --> I run a current CS6 setup on MAC and PC with MPE enabled and Quadro board on MAC - GTX on Win7...

                                   

                                  1. Realtime H.264 encoding?? Would like to read more on this - maybe for the Youtube preset but if you do a propper twopass encoding for 1080p24 Bluray and render from DPX files it's hard to manage...)

                                   

                                  If I encode a 90min feature with AME to M4V Bluray from 8Bit Quicktime it takes about 4.5h on a quad core i7 3.6GHz and about 3h on a dual quad core Xeon MAC 2.4GHz)

                                   

                                  If I encode from DXP render times doubles - Raid is reading just 80MB/s - quad core i7 3.6GHz is just at 75% load - and 90min encoding typically takes about 8h...

                                   

                                  2. If I activate Maximum Depth and Max Render Quality and export to DPX - Raid hasn't load - CPU hasn't load - GPU hasn't load - sytem is idle - but the export takes forever...

                                   

                                  3. If I activate Maxiumum Depth on export only half the CPU cores are used for rendring intense tasks like JPEG2000 encoding - If I export with 8Bit all cores are used imediately. (tested with single socket quad Core i7 3.6GHz and dual socket quad core XEON 2.4GHz)

                                   

                                   

                                  so - fast Hardware alone doesn't results in fast render times...

                                   

                                  1. Even on the most recent 8 or 12 core systems you can be happy to get realtime but not much faster (has anybody testet GPU accelerated Bluray compliant HQ encoding from AME??)

                                   

                                  2. If DPX files are part of the workflow things get slow. (fast Raid or SSD helps but not much)

                                   

                                  3. If you activate Maxiumum Deph things will take 4x to 10x as long as with 8Bit - at the same time half of your CPU cores are set to idle - doing nothing anymore

                                   

                                  4. Using 32bit Quicktime + badly multitheraded codecs via IP link seems to slow down PP - If you read and write MOV files rendering takes longer than on native PP formats. (But in real live you get Prores HQ as source files and have to deliver Prores or DNXHD MOV back... so it doen't helps that PP can export MPEG II blazing fast

                                   

                                  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

                                   

                                  so what speed PP up and whats the bottlenecks?

                                   

                                  1. GPU for MPE - mandatory don't try to live without

                                  2. fast SSD for temp files (media cache) - mandatory don't try to live without

                                  3. fast RAID and SSD's for footage - esp. on I/O intensive tasks like DPX file lists SSD's are helpful - if the data becomes too big a fast HDD Raid can help (If your Raid can manage 600MB/s be happy if Premeier uses 150MB/s

                                  4. Fast CPU or dual socket system: currently I only get my quad core i7 3.6GHz with H.264 encoding to 100% load - if I work with DPX files I can't even saturate one CPU with load... before you think of a dual socket system by some PCI-Express SSD-Cards first ;-)

                                   

                                   

                                  what will slow down your renderings massively?

                                   

                                  1. Usage of DPX files (regardless of the speed of your Raid)

                                  2. Usage of Maximum Depth on export

                                  3. Usage of Quicktime

                                   

                                   

                                  Any best practice suggestions?