7 Replies Latest reply on Aug 6, 2012 7:01 AM by Jim Curtis

    Stability - At Last(Long Post)

    AndyJenkins39 Level 1

      I hope this thread finds you well.


      Quick back story, and then hopefully some useful information:


      About 4 months ago, I made the decision to use Premier Pro on a rather large and complicated Feature Film.  During production, there were 3 rigs going full time - an Ingest and Encode, a Sync and Dailies, and the Co-Editors Rig, who was cutting a coverage edit to make sure we were capturing what we needed.


      All was well.  Editorial was always just about 24 hours behind production, a testimate to not only PPro's robustness, but also our Terrific DI who kept the machines fed with media from multiple sources.


      This is probably a good point to mention that we were on CS 5.5.2.


      About 3/4 of the way through the show, 6.0 was released.  My co-Editor, being a fan of the new interface along with having warp stabilizer right in the timeline decided to pull the trigger and migrate the post.


      Of course, we kept all of the daily log projects (Which contained out Sync'd files, Dailies and One-Lights or R3D RED files) in a CS 5.5.2 version and continued Ingest and Sync in 5.5.2.  The Coverage Edit moved into 6.0.


      A few weeks later, production wrapped and moved into post production in Southern California.


      That's where it got Ugly.


      We experienced just about every issues that's been repeated in this forum. For example:


      • Random Serious Error Crashes
      • Timelines that would not re-draw
      • Sequences that would not export (Out of AME or PPro)
      • Random Dropouts of Audio, followed by an absence of Video Playback
      • Repeatable Errors - I.e. Serious Error Crashes whenever an extract edit would intersect a dissolve on an audio track. That makes trimming out a shot... challenging.
      • Every Exit would Require a Force Quit.
      • Randomly Corrupted Projects (With an inconsistent Auto Save, Time Machine was critical to recovery)
      • etc.


      About a week ago, the film made it to rough-cut stage, and was given over to the producers and director for a notes pass.  That left the co-editor and I about a week to start a process of troubleshooting with the goal of creating a stable rig for the rest of the editorial process.


      After 4 solid, documented days, we have arrived at what has demonstrated to be the most stable implementation of PPro 6.0.1 that we've had.


      Following is the process in exact order - I hope that it helps if you're encountering frequent issues.


      1 - Fresh Reinstall of Mac OSX 10.7.4 

      Wipe the hard drive, reinstall the OS, and update the OS until no further updates are available.


      2 - Repair Permissions on the OS drive.


      2.5 - Install any peripheral hardware.  IMPORTANT - The only additional hardware on this "Hero Rig" was a 2 Channel eSata card.


      3 - Configure Media Drive.  In our case, we are running a 4 Disk Raid 0 - That's 4 3GB 7200 RPM Enterprise (Meaning, NOT GREEN) drives.  This is a SoftRaid powered by SoftRaid 4.3


      4 - Copy Show Media to Internal Media Raid - we have 6 discreet back ups of the media in 3 different locations on external Raid 5 drives.  This copy took about a day, given the 5 TB of data used in the show.


      5 - Install Premiere Suite - Allow Java Update.


      6 - Repair Permissions on OS drive.  This turned out to be critical, as the moment we installed the CS6 suite, permissions on the drive were reporting errors.


      7 - Install Digital Rebellions Maintenance Tool - Do a full Housekeeper, QuickFix, & CS Repair.


      8 - We use DropBox to keep a Cloud Back up of all of our projects - so, install DropBox and restore projects to OS Drive.


      9 - Begin Time Machine Back Up of Entire System.  To do this, we are backing up to a 4 Drive (4x4GB) RAID 5 External Drive by OWC connected via eSata.


      This Initial Back Up took about 8 hours.


      10 - Clone OS drive to a discreet 7200 RPM 1 TB Drive and Store nearby.


      Here is our Hero Rig Configuration:


      Mac Pro 5.1

      OSX 10.7.4 (Tried Mountain Lion, but for reasons below, reverted to Lion)

      64 GB Internal Ram

      480 GB Accelsior PCI SSD for OS

      4 x 3GB 7200RPM RAID 0 for 12TB

      2 Apple 30inch Cinema Displays

      Audio Playback is via Digital Optical Out to an external amplifier attached to Genlec monitors in a 2.1 configuration. (Note, we feel this has contributed to further stability, by not asking for additional audio processing through 3rd party hardware, and instead relied on the System Level Optical Out and allowed for processing outside of the rig)


      And the final piece to the puzzle - perhaps the MOST important item that contributed the MOST to the stability of the RIG...




      I'm almost positive that you hate that last part.  Trust me, I understand.  We had brand new Quadro 4000's in all of the editorial rigs, and one rig with multiple Quadros' in order to run Da Vinci's Resolve.   They are now in a single rig that we have designated for Color Correction and Conforming.


      All of the editorial rigs are powered by ATI Radeon HD 5870's.


      Please believe me when I say that this was the last item we tried to remove from the configuration.  Yes, we attempted to work with the configurations with GPU acceleration ON and OFF, and no matter what the state of acceleration, the Nvidia Drive and CUDA drivers contributed the MOST to the systems instability.


      We tested with 3 Quadro 4000's to ensure that we didn't have a random bad card as well.


      As a further precaution, on all but our finishing rig, we have removed all 3rd party plug-ins (Magic Bullet Suites, Colorista, etc.) as well as our BlackMagic and AJA I/O cards.


      What we are left with is a set of rigs that will run the day without a crash, so long as we do NOT try to do an extract edit across a clip that has an audio dissolve, and we navigate around the timeline with all of the audio tracks configured with the waveform display OFF.


      We are working with 1080p ProRes Media at 23.98 and audio is 48K 16bit BWFs (Broadcast Wave Files)


      Some Considerations:


      Our Timelines are complex.  We're cutting an action film with lots of quick cutting, speed ramps, trick dissolves, etc., and in preparation for Sundance Submission, we have an average 35 tracks of Temp Sound FX and Music.  Our Production sound is an average of 8 channels of audio as well, making our average timeline contain about 42 tracks of Audio with about 6 tracks of Video.


      Our Projects are large.  The average shoot day contained about 250GB of Media broken up into about 20 scenes each with multiple takes from multiple cameras.  A single REEL might then contain 8 days of source media for about 2 Terrabytes of production assets.  In REEL 1, for example, there are about 6,000 pieces of media loaded every time we open that project. REEL 1 was approximately 275MB in project size. We currently have 6 REELS comprising the Rough Cut.


      I'm simply pointing out that the nature of our project has caused us to run as lean as possible in order to maintain stability.  Your situation may not resemble ours in terms of project and timeline complexity, so you may find yourself able to run a rig with more accessories.  This just happens to be what we're using to accomplish our goal of a stable production environment.


      Finally, I remain convinced that any forward compatibility does leave some sort of remnant code in the projects that causes a degradation of performance over time.  Meaning, we're now creating Master Projects that contain all of the logging information that we require, as well as creating Leaner Editorial "Working" Projects where all of the unused media is removed, as well as clearing out old versions of sequences we no longer need.  These new projects are created in 6.0.1 and appear to be more reliable than a 5.5.2 project that is up-converted to 6.0.1.


      In retrospect, I should have insisted that we remain on 5.5.2 for the balance of this show. The transition was indeed very frustrating, but for the record:


      I was raised on Avid Media Composer since its release in the late 80's.  NYU (My alma mater) had one of the first installs in the country, and I learned the interface there, and won my first Emmy on an Avid Rig. I've since cut on Symphony, DS, Flame, Inferno, etc.


      I've completed several feature films on FCP v7 - including 6 that you might have seen theatrically and can currently buy on iTunes, etc. And yet...


      Aside from sharpening some trimming functions, adding match frame to the source side of the monitor, and allowing for L-Cuts without having to unlink synced media...


      ... I am very happy to be cutting this show on PPro.


      I hope you are able to achieve the same result.




      Andy Jenkins


      Producer/Partner/Editor - Haxan Films

      Creators of The Blair Witch Project



      P.S.  I am in no way affiliated with Adobe, nor compensated by them.  The licenses we own of their software were purchased via retail channels.


        • 1. Re: Stability - At Last(Long Post)
          shooternz Level 6

          Congratulations on your achievement but also..thank you for the time and effort you put into your well considered  post.


          It is very refreshing to hear from the leading edge instead of the bleeding edge.



          As regards


          Aside from sharpening some trimming functions, adding match frame to the source side of the monitor, and allowing for L-Cuts without having to unlink synced media...


          If you want to start a separate discussion thread.. I can contribute at least  one thing that may be helpful to you in these matters.



          • 2. Re: Stability - At Last(Long Post)
            lasvideo Level 4

            Its a good idea NOT to upgrade in the middle of a project.


            Ive been running CUDA cards on a Mac (Quadro 4000) and PC (GTX 570) on graphic intensive projects with no issues like you experienced. Inital use was with CS5. I am currently on CS6. I did not upgrade in the middle of any of these projects which probably accounts for the stability I experienced.

            • 3. Re: Stability - At Last(Long Post)
              Harm Millaard Level 7

              I second Graig's reply. The only point of criticism is that you should have mentioned MAC earlier in the post, because then I could have skipped it completely.

              • 4. Re: Stability - At Last(Long Post)
                Level 4


                Thanks for this post about your hardware and software setup and also the workflow aspects...backups and so on...

                I found that very useful as I tried to imagine exactly what's going on , when , where, and how fast it had to happen.

                I'm not surprised you had ( have ) so many audio tracks, so your description of timeline and basic vague info about multiple sources and cameras and takes etc sorta helped me envision what you've done. Congratulations on your work and keep it up... I hope you are as in love with it as you were when you were at NYU.


                About the time you were at NYU I worked on something called 'death mask' which was a pretty low budget movie in nyc area.



                Yuri Denysenko DP


                Yuri taught at NYU and maybe you knew him ?


                One of the coolest things about this movie was being around farly granger (sp? ). He was a really down to earth man who didn't have a big head or any " airs " about him and told us stories about working with Hitchcock etc that were fun to hear...and also instructive.


                Anyway, your post brought this memory back to me ( even though they mispelled my name on credits .. its amazing they actually finished this movie at all now that I think about it ).


                Good luck and thanks again... if you can share more work flow type posts in the future, the lounge is a cool place to just yap and talk about all sorts of things... like when its beer thirty...



                • 5. Re: Stability - At Last(Long Post)
                  JAKE JONSON Level 1

                  Andy, thanks for your well thought out post.  I love hearing stories from the trenches, and particularly when it ends happily.  Thanks again, although I too can ignore the Mac-specific issues since I'm on a Win 7 platform.

                  • 6. Re: Stability - At Last(Long Post)
                    Level 4

                    I have an OS and platform that I built myself and it isnt a mac and it isnt a PC.  It is something I built all on my own.


                    I use Premiere Pro and love it.


                    I won't tell you what I am using because you are probably using a mac or a PC.  So it is a waste of time to talk about my fanstastic invention at this point...but in a week or two I will be getting my copyright and marketing my new computer hardware and OS so that the whole world will be really really impressed !


                    Meantime, there is the Premiere Pro adobe software this forum is about, rather than the hardware... and I am guessing there's room for everyone to get into the software a little bit ??  The workflow and the time and so on it takes to work for real on projects that depend on that ?


                    I just don't get the posts about " if I knew this was mac I would have ignored your post ".  What's up with that ?  Who gives a hoot what platform it's on ?  I dont get it.


                    It is now beer thirty ! Relax and make movies and live a little bit... tell stories, have fun... help one another be silly....enjoy life !



                    • 7. Re: Stability - At Last(Long Post)
                      Jim Curtis Level 3

                      I also want to thank you for the post, the details, and your conclusions.  Food for thought.