18 Replies Latest reply on Aug 6, 2011 4:33 PM by JDtogs

    AME and Premiere CS5 + Cineform HD - very slow HD encoding

    JDtogs

      Hi all

      This is my 1st post here, it is rather lenghty but I will try to give as much detail and hope someone can shed some light on my issue.

      Also, please comment if a better worlkflow can be recommended.

      I'm using PP CS5.03, Win 7 SP1 64bit, i7-2600K @4.6Ghz, 16Gig DDR3 1600, GTX580 - Mercury enabled in PP, 3disk 7200RPM Raid0 array for source footage and scratch disk, a 2disk 7200RPM Raid0 array for exported files.

       

      My encoding seems to take forever as AME only uses 30% of my CPU, 7gigs of ram is currently being used by the entire system for the encoding - i'm sure this is not right, i have tried every other setting with no luck. When i use AME on its own to convert raw footage to a different format it use 90-100% CPU and rendering is done in no time.

      I have a very long film 3hrs:32mins that I need to export to mpeg2 HD 1920x1080 25fps. (I need to make a preview file for the client that will take the least time to encode with reasonble "small" file size - estimate size for this project is 24GB) - other reccommendations with different formats, similar quality, small file size and minimum encoding time will be appreciated - must also be playable on PS3)

       

      The source footage was 25fps shot with 4 5DMkii  cameras, all footage was converted to Cineform HD codec, 10bit uncompressed.

      PP sequence is Cineform 1080P with Max Bit and Max Render enabled (I always turn these on). I prefer to use the Cineform codec as loading large projects is very quick; near instant on my system opposed to using the native MOV 5D files - PP always seems to hang with the MOV files and when I clicked outside of the PP work area and back PP hangs and there's a waiting period - very annoying; since using the Cineform codec this issue is gone.

      Plenty of the footage was fixed in AE - stabilizing, color corection, etc.. and rendered to Cineform HD codec to match my project. Rendered files was imported back into PP and linked footage was replaced with the AE rendered clip. All other effects on timeline has been rendered in PP, timeline bar displays green and yelow.

      When I export my project using Media Encoder 5.01 it does not use all the system ressources, 30-40% CPU , see screenshot here with a 80% spike now and then and drops down to 30% again, (don't think this is normal either) 7-8GB ram max, I also noticed that my GPU has a 20% load during export. I don't mind waiting 3hrs or less to render but 10hrs +?? Can it be the Cineform codec slowing things down? I NEED this CPU to run at max to speed up this process, any help? On short test I've done from PP to AME with the smae settings the render is very fast when the CPU is fully loaded.  Am I doing something wrong with this large project?

        • 1. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
          JDtogs Level 1

          hi, the final render took 6hrs, this should have been much quicker on my system.

          I did another test after the render was done using clips from the same project - 2mins of the MOV to Cineform HD codec and 2 mins of AE HD Cineform rendered files, the 4 mins took only 1min 33secs to complete with maximum render quality enabled and the CPU was running at 70-85%. This is more or less what i'm expecting on a full export render, but its not the case. I also rendered the enitire timeline (green bar) but still no improvement on overall export render time - has anyone expereinced this before?

          • 2. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
            Bill Engeler Level 2

            Rendering the timeline prior to export has no effect on the final export.   You didn't mention the effects that have been applied to the timeline - these can make a big difference.  2x realtime encoding isn't very unreasonable with a few non-CUDA effects, like Colorista or Looks.

            • 3. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
              MobyTrix Level 1

              JDtogs wrote:

               

              I also rendered the enitire timeline (green bar) but still no improvement on overall export render time - has anyone expereinced this before?

               

              Preview files won't be used for any export unless you check the checkbox "Use Preview Files" in the export dialog (it's at the bottom).

               

              Getting a 3.5 hour 1080 HD project to render in 6 hours seems pretty reasonable to me, but it does seem odd that your machine is not utilizing all cores.  There's only two things that would cause that, and both are I/O-related:

               

              - Effects are being offloaded to the GPU inefficiently, slowing the process down

              - Your hard disks are extraordinarily slow

               

              Try running some general purpose benchmarks on your system to make sure that CPU, memory, and hard disks are all performing to your expectations outside of Premiere; SiSoft Sandra is a good starting point.  (benchmarks take over 15 minutes so be patient).

               

              Also, the 7G RAM memory usage is unfortunately par for the course -- I get that too when doing Encore builds.  With 12G of RAM I thought I didn't need much of a swap file, but encore+PProHeadless.exe were gobbling everything up, so I just created swap = RAM (12G RAM, 12G swap) and crashing/performance issues went away.

               

              Finally, if you're overclocking, take it back to default settings.  Overclocking incorrectly can speed the CPU but slow down other components.

              • 4. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                JDtogs Level 1

                I used levels and Looks on several clips, I know that Looks is non-Cuda  and will take a bit longer to render, just dont understand why PC is not using more resources to speed up the process. For some reason my short test sequence renders very fast with the the exact same sequence and export settings (same files as project with filters as well) and it utilizes most of the available resources.

                • 5. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                  JDtogs Level 1

                  "Use Preview Files" is always checked.

                   

                  My 3 X 1TB SATA6 disk are in a RAID 0 config - HDTune Pro - 405MB/s read/write, this is a pretty descent bencmark score, I don't expect to get the same speed during render but for some reason my I/O speeds are extremely slow - monitored this with windows resource manager and there's definitely something funny happening there. Any advice where to look/solve this issue?

                   

                  I have done plenty of benchmarks scores and test outside Premiere with good to very good results, enitire PC stress test up to 12hrs, stable and low temps, no issues.All my other apps runs very smoothly, quick and efficiently. After effects is running like a dream, even while rendering the Looks filter on AE projects files, renders are realtime or faster. Only when I load Premiere and Media Encoder for this large project then performance drop big time.

                   

                  I agree, 6hrs for a fullHD 3.5hr video render is not bad but this CAN be improved.

                   

                  Can you give me a run down on creating that swap file? Would be keen to see if that will make any changes.

                   

                  Overclocking - this is correctly done with more then 12hrs of stress testing - i've tried Premiere and Media Encoder with OC on and off, slower with overall render, OC off and fast to very fast with smaller projects with the OC setting on. Is Premiere maybe the culprit struggling wit this large project? I have little to no performance issues when exporting smaller sequences.

                   

                  My next step will be to break up this project into smaller sequences and export and see if there's any changes in overall render time.

                  I have a few lenghty HD projects coming up soon so I will see if I need to make some changes to my workflow as well.

                  • 6. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                    jkosmicki Level 2

                    You may want to try a sample export with "use maximum render quality" turned off in AME. Doing that seems to speed up my exports 2X ( from 10min down to under 5min). When I've done that I have not seen any loss in quality, but I don't really do any effects in Premiere. I almost always do all of my effects in AE and render to CineForm for import into Premiere.

                     

                    It may be worth a try to see if, A) render times improve and, B) if there is a visable difference in quality of the output.

                    • 7. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                      JDtogs Level 1

                      I have tried this sample export way in the beginning already, 4min clip took 1min:33secs with Max render off and 3mins with Max render on. I agree that I theres hardly any noticeable quality loss. My main project has Max Render turned off and the results is much slower then the test project. Has anyone had great feedback with rendring a large HD project (3hrs + ) using Premiere and AME?

                      I will run the PPBM5 benchmark test later and will give some feedback on that.

                      • 8. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                        digitlman Level 1

                        My experience in premiere with low cpu usage is usually tied to using a filter or effect that is not multithreaded to use all your cores so it slows everything down.

                         

                        if a raw file with no effect renders full speed then i think i am on the right track. you could render out a small test section of your project and disable any filters/effects there and look at the difference.

                        • 9. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                          MobyTrix Level 1

                          digitlman wrote:

                           

                          My experience in premiere with low cpu usage is usually tied to using a filter or effect that is not multithreaded to use all your cores so it slows everything down.

                           

                          This is a brilliant observation and is most likely the OP's issue.

                           

                          OP, as a test, try making a 1-minute clip and placing it several times in the timeline, then to each section, apply only one of the filters you have in your real project.  Render Work Area and then see which section takes the longest time -- that will point you to the effect/filter slowing you down.

                          1 person found this helpful
                          • 10. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                            JDtogs Level 1

                            that sounds like a good plan, i will be in front of the pc this afternoon and start with some more testing. Will follow your advice and keep you posted. Thanks all for your input thus far.

                            • 11. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                              JDtogs Level 1

                              My test results - 5min clip exporting to mpeg2 HD 1920x1080:

                               

                              using 5x 1min clips from original project (Cineform HD 1920x1080) with MB Looks filter, Fast Color Corrector, Color Balance, Cross dissolve

                              Export the project without rendering the work area took 8:27secs - Max Render Quality - ON

                              Render Entire work area.  export with Max Render Quality - ON, Use Previews - ON - took 2:03secs

                              I exported to all my drives with small difference in rendering times

                              2 disk Raid0 + OS  - 2:13

                              3 disk Raid0 + source files - 2:03

                              2TB backup drive - 2:23

                               

                              CPU load is 75-90% rendering standard transitions and filters - GPU load is 0%

                              CPU load is 80-90% rendering MB Looks filters - GPU load is 0% (i use this filter quite often)

                              CPU load is 75-90% rendering clip with Fast color Corrector filter - GPU load is 28% (i use this filter quite often)

                              Max RAM usage was 9.4GB  - would like to see more of these being used - 16GB installed - 13GB shared between AME, AE and PP,

                              Premiere CS5 was closed during testing, no other apps in background.

                               

                              These test results shows that my PC is running not to bad with very acceptable render speeds - for small projects - 2.3X faster then real time

                              Why can't I achieve the same rendering speeds with my larger project? This really baffles me. 6hr renders might be OK for most users but based on my test and PC setup I should be able to achieve much better results when rendering the full project.

                              • 12. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                                MobyTrix Level 1

                                When I render out my 3.5-hour project, I have noticed (by watching Process Explorer's graphs) that AME will "scan" all of my assets, presumably to make sure none of them changed or that the audio still conforms (or maybe that's when it conforms the audio).  I don't know why it does this, but when the scanning is done, THEN my CPU lights up like a christmas tree and the actual work begins.  This initial scanning can eat up 10-15% of the total time, and slower drives would compound it.  I currently have all assets on a 2-drive RAID-0 with a max read speed of 160MB/s.

                                 

                                For your longform project, what target are you rendering it to when it takes so long?  A compressed target like H.264, or Cineform?  If a compressed target, that is eating up more time than is comfortable (AME is a slow compressor compared to others I have used).  I haven't worked with Cineform in a few years, so I can't remember if it's very fast at compressing, but I seem to recall it was.

                                 

                                If you have a REALLY fast I/O subsystem, like a RAID-0 with 3 or more drives (I seem to recall your rig was capable of 400MB/s), have you tried rendering to an uncompressed format like UYVY (should be in the Uncompressed AVI group)?  And was that any better?

                                • 13. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                                  JDtogs Level 1

                                   

                                  When I render out my 3.5-hour project, I have noticed (by watching Process Explorer's graphs) that AME will "scan" all of my assets, presumably to make sure none of them changed or that the audio still conforms (or maybe that's when it conforms the audio).  I don't know why it does this, but when the scanning is done, THEN my CPU lights up like a christmas tree and the actual work begins.  This initial scanning can eat up 10-15% of the total time, and slower drives would compound it.

                                   

                                  I noticed the audio conforming process in the beginning which takes up about 6-7mins of the rendering process and CPU processing only starts after that - I agree and i'm fine with that.

                                   

                                  If you have a REALLY fast I/O subsystem, like a RAID-0 with 3 or more drives (I seem to recall your rig was capable of 400MB/s), have you tried rendering to an uncompressed format like UYVY (should be in the Uncompressed AVI group)?  And was that any better?

                                   

                                  I have rendered to uncompressed HD formats before; AVI, Cineform - but with ridiculously large file sizes - 5mins footage will be something like 14-15GB. I dont see the point in that for my final output. I will do a test with the UYVY format and see what results I can come up with.

                                  My RAID 0 setup is pretty fast at the moment, this will be upgraded later to 3X Vertex3 SSD's in RAID-0 for some serious I/O speeds.

                                   

                                  I use the latest version of the CineformHD codec, which is not slow at all, but I only convert my RAW files to this for easier workflow and export to mpeg2 HD and h.264 for my BluRay projects.

                                   

                                  Anyway, I "think" I might have found the problem to my large project slow render - maybe I was at fault, but don't know, please advise if you think my editing skills need some fine tuning...

                                   

                                  We shoot with 4 cameras, so I start with 4 video tracks and up to 10 audio tracks for external sound.

                                  I just deleted the extra unused video clips on the timeline which was either turned off via "enable" option or i had a video clip placed above an unused clip.

                                  Removing the unused clips required me to re-render certain clips or effects that has been rendered before - I just rendered the first 15mins of the timeline and noticed a HUGE improvement when I exported using the same preset as before.

                                   

                                  before: 15mins of the project took 32mins to render including the 7mins audio conforming in the beginning. (this was before i made the project changes)

                                  now: i added the sequence to AME, closed PP and free up the ram that PP was using and started the AME quee - 15mins of the the total project now only took 19mins including the 7mins audio conforming. That means the video rendering took 12 mins to render 15minutes of footage - 3.5hrs of footage should take just under 3hrs to render if everything runs smoothly. I'm currently rendering the rest of the timeline previews and will export the entire project after that. Will post my results  - at least i'm smiling for now

                                   

                                  Question - could the unused footage on the timeline have been the cause for my longform slow render?

                                  Is it standard practice to just remove unused clips from the timeline? I  have always just disabled the unused clips and never really noticed any issues with smaller projects. I guess I will be making changes to my workflow in the future.

                                  • 14. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                                    Bill Gehrke Most Valuable Participant

                                    When you say Export the Project using AME, are you exporting from Premiere or are you opening the project/sequence in AME directly?  It makes a major difference in performance If you open directly from AME you are using the assets from you fast project disk.  If you are exporting from Premiere and have not changed the caching location you are then using the default slower Boot drive location cache files

                                    • 15. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                                      JDtogs Level 1

                                      Bill, I export from Premiere to AME. My media cache location is on my faster 3 disk RAID 0 array.

                                      I have not tried importing the sequence directly from AME yet, will do so when the current render is done and see if there is a performance increase.

                                      • 16. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                                        JDtogs Level 1

                                        I have some pretty good results to share and I think I now have the solution to my initial issue.

                                        After a further 12hr rendering and testing session I ended up with these results.

                                         

                                        3.5hr 1080P project export from PP to AME, Maximum Render - On, Use Previews - On, all effects on timeline rendered.

                                        CPU usage was between 75-100% and drop to 35-40% for transition effects, MB Looks filters was rendering at a steady pace 60-70% CPU

                                        Max RAM used - 13GB (16GB installed, 3GB reserved for other apps)

                                        1080P export to 3 disk RAID 0 array + source files,  -- 2hr:45mins

                                        1080P export to 2 disk RAID 0 array + OS,  -- 2hr:51mins

                                        1080P downscale to 720P export to 3 disk RAID 0 array + source files,  -- 5hr:50mins

                                         

                                        I noticed no improvement importing the sequence directly from AME (PP is closed) as was mentioned before, performance was similar when exporting from PP to AME.

                                        I am very pleased with these results - from 6hrs to 2hrs:45mins is definitely a huge improvement for me. The system was very stable during these renders, I could use it as per normal (although i just leave it when it renders), all available RAM was used, max GPU load during render was 47%, CPU temps was running cool at 68C @4.6GHz. (who said an overclocked CPU is not good for video rendering??) Now it's time do to some cleanup and move on to the next project

                                         

                                        I knew the system was optimized for CS5 before i came to the Forum, just could not understand why the long rendering time, and now I know why; deleting unused clips from the timeline for these long-form projects will be number 1 on my list before I render the timeline for final Export. This was a valuable lesson and thanks to everyone for their input.

                                         

                                        Edit: I also changed Premiere settings to Optimize for Performance instead of Memory. Don't know if this changed anything as all the rendering was done via AME.

                                        • 17. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                                          Toomany3 Level 1

                                          JDtoqs, I know this post is several months old now, but do you have any additional observations?  Your situation sounds very much like mine.

                                           

                                          I will try deleting unused layers and see if that helps.  I also have several monster machines and I notice that on longer projects, the renders come to a crawl.

                                          • 18. Re: AME and Premiere CS5 + Cineform HD - very slow HD encoding
                                            JDtogs Level 1

                                            Toomany, you've probably noticed that I have'nt posted in a long time, that's because my rig is running very smoothly since these issues mentioned above.

                                            The only thing that really bothered me was the increasing project file size on my larger projects. The project loads slowly, takes longer to save, crash without error screen when I try to export... that was with PP CS5.03.

                                            That particular project folder was 800GB including all AE renders and media, the PP project file was 125 MB with 12 sequences.

                                            I'm using CS5.5 now, my new project folder is 720 GB and PP project file is 58 MB.

                                            Loading the project now takes roughly 5-6 seconds and saves are near instant - the way it should be.

                                            I mostly use MBL 2 on my timelines so rendering is a bit slow but it does not take forever and of coz my renders are still fast on this PC if I dont use any MBL filters. 5min clip , Cineform Codec to Cineform.avi 1080p takes 6 mins to render and 5-6 times longer with MBL 2 filters on the same timeline.

                                             

                                            I have no issues with the Cineform codec. Upgraded to the new 5.5.2 version last week, it's said to be a bit more stable in PP.