25 Replies Latest reply on Dec 11, 2010 5:01 PM by hhansard

    So PP CS5 does NOT really allow for thousands of clips, eh? Advice...

    Videowilk Level 1

      It appears despite all the press about the Mercury Playback Engine allowing Premiere Pro CS5 to handle projects with "thousands of clips", it is UNTRUE.

      I have had very few problems since moving to CS5 (without Matrox Axio drivers thus far, after some issues on CS4). So, up until now, I've been extremely happy and productive with CS5.

       

      Fully certified system Adobe Production Premium CS 5.02

      Windows 7 64-bit

      Nvidia 4800 CUDA - 197.90 driver; QT version 7.65.17.80 - carefully stayed with these versions as directed by Lyn Nortsad who built system & Adobe Forum

      24 gb RAM

      8 TB RAID 6 SCSI

       

      Yes, I have a huge project with tons of clips (6,500) shot with Sony EX-3 (mxf).It's tons of interview clips and cover footage. I started out by importing the interviews, culling them down working to build the basic storylines with the plan to then import the cover footage. It was fine with the interviews - initially taking an hour or two to open the project. So I would leave it open overnight.My trouble started once I imported the cover footage... white screens and crashing with a couple of  MXF_SDK_r.4.2.2.319.dll errors in the Event Viewer. Now when I open the project, there are 6,274 things in the lower left hand corner of the screen loading. Yet once that finally finishes - hours!--- there are many unlinked pieces of media throughout the entire project. So now it crashes when I try to link a large group of files back. This is annoying to say the least. So I know I need to shrink the number of clips and that Adobe was untruthful in their claims about working with long form projects with thousands of clips. They must have meant only 2 thousand or fewer.

       

      I got the suggestion of dividing up the project into several projects... but how do I do that if after creating a New Project I can only import the entire gigantic Project (which would seem to lead me back to where I am) or selected Sequences from it? I wish it offerred me the choice of also importing selected Bins, since that's how I had organized the initial footage. I'd appreciate any advice on how folks can really work with "thousands of clips" in Premiere.

       

      Also, would changing any Preferences within the Project help? I fear deleting the Metadata Cache would mean recreating all of the PEK and CFA files, thus bogging things down once again. I've also read something about changing the XMP writing in Preferences. I do not understand how many clips become "unlinked" upon reopening the project. I do not move them around, they all sit in a folder on my array. I believe I have the correct workflow for working with the EX-3.

       

      I am now going back to Sony Clip Browser on another computer to go through original EX-3 files and select fewer cover shots; then will copy those to the RAID array and import them. This is very time consuming - since I had hoped I could cull through and organize all the footage within Premiere, building sequences related to each section of my overall program, while also deleting clips I didn't think I'd use. Of course I appear to have the time now that I am waiting hours for the project to open.

       

      I am now down to 4,364 clips remaining to be loaded... 45 minutes later, and seeing many unlinked clips in the bin...

       

      Advice? Am I the only person to have 6,000 clips in a project?

       

      Happy Thanksgiving Editing Day,

      Diane

        • 1. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
          Harm Millaard Level 7

          From the sound of it, the system may be slightly overwhelmed by the amount of data.

           

          8 TB SCSI Raid6: Is that a 20 x 450 G Raid6 array? On what controller? Is that local storage?

           

          What CPU and clock speed?

           

          What OS disk and other drives?

           

          Why use that far outdated and old nVidia driver?

           

          as directed by Lyn Nortsad who built system & Adobe Forum

           

          I'm not sure that pleads for him. The Jive software should never have been used in the first place.

          • 2. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
            hhansard Level 1

            I had similar problem experimenting with time-lapse photography (about 6000 HD stills or “clips”).  The only way to get this to work was to combine every 256 stills (or so) and encode a series of lossless avi intermediates.  The final project was combining (about) 24 avi files into a final encode.

            I am sure there is a very high technical limit on well-tuned high-end hardware; I bet the practical limit is far lower.   Will be interested in others experiences.

            • 3. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
              Videowilk Level 1

              Hi Harm - It's an HPxw8600 with Dual Intel 3.4 GHz Xeon Quad-core CPUs with 12MB cache and 1333 MHz FSB andATTO/LSI U320 SCSI Host Adapter with an Enhance RAID T8 (8x1TB) array. 500gb system drive and the arrays are striped and local. All media is on one set of 4 drives, local.  I had previously updated Nvidia drivers and had issues, so that's why I went back.Again, all was fine with CS5 until this humongous project.

               

              I don't understand the Jive software reference - do you mean the Clip Browser? I have always used the Clip Browser to convert to mxf for import into Premiere. Do you have another suggestion?I heard importing directly from BPAV folders creates additional steps in Premiere.

               

              I'd like to know why Premiere takes 876 clips and turns them into 37,972 files (klv, pek and assoc.xml)? I think that's the issue. So I am doing my best to find another way to cull through footage and lessen the number of clips. But I thought CS5 and MPE would allow for such a large project.

               

              -Diane

              • 4. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                Harm Millaard Level 7

                Diane,

                 

                You have me confused. First you said:

                 

                8 TB RAID 6 SCSI

                 

                and now you say:

                 

                the arrays are striped and local. All media is on one set of 4 drives, local.

                 

                This is conflicting. It either is a single level parity raid, or if there are more arrays defined a multilevel raid if they are striped. So, what is your array configuration? And which type of Xeon's, because most are clocked lower than 3.4 GHz, unless it is an older model.

                 

                Jive is the software used for the Adobe forums that causes problems with a regularity like clockwork.

                 

                Importing complete BPAV directories is the preferred way, because the chance that metadata or links get lost is far smaller than importing all by hand. Just importing a complete directory is a lot easier and faster and more reliable than importing individual clips. The only extra step is that you have to disregard the error message about unsupported files.

                 

                The extra files created are the CFA and PEK files needed for editing the audio parts and displaying the waveforms in a non-destructive way.

                • 5. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                  Videowilk Level 1

                  Hey again - Sorry for confusing info - processor should be Xeon x5450 3.0 (dual quad core). The array is set up as 2 sets of four 1TB dives (8tb total) but with RAID I guess 5, not 6 so that it is really 2 sets of 2.72TB drives striped together with RAID protection. I called them local because they are my D and E drives, where all the media for all my projects is stored.

                   

                  Are you saying the BPAV folder should be copied to my media drive, then imported using Premiere? I thought that would mean having to go into each individual clip folder to import/transcode the actual clip. So I always understood using Clip Browser eliminates that extra step and creates easy-to-import mxf files. I did not know you can import a entire Folder from within Premiere... I'd love to hear how others work with large numbers of EX3 files.

                   

                  Anyway, I'd still like to know how many clips is too many by Adobe standards - Microsoft tells you the file size limitation of your personal mailbox folder- why can't Adobe say ### clips are too many?

                   

                  How could 876 mxf files turn into 37,972 files? Those include the pek, klv and xmls.

                   

                  Should I change Preferences - to uncheck the Write xmp data to files on import? Would that speed things up?

                   

                  If I clean out the Media Cache Database, I fear it would do the audio conform on each clip all over again -taking another several hours to load project.

                   

                  So I am continuing to go back to EX3 files in Clip Browser, selecting fewer clips to then import into Premiere. Any advice on workflow for my dilemma is greatly appreciated!

                   

                  -Diane

                  • 6. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                    Harm Millaard Level 7

                    Diane,

                     

                    The array is set up as 2 sets of four 1TB dives (8tb total) but with RAID I guess 5, not 6 so that it is really 2 sets of 2.72TB drives striped together with RAID protection. I called them local because they are my D and E drives, where all the media for all my projects is stored.

                     

                    That is not possible. Either they are striped and appear to Windows as one large disk in a Raid 50 array, or there are two separate Raid5 arrays, seen as D and E but are not striped. Based on your info I assume you have two 4 disk raid5 arrays, not striped. Since you say you have your media on D and E, take care of nicely distributing those media across both arrays. The alternative is to indeed stripe them to a Raid50, which will give you one giant 6 TB disk, and seen as only a single disk. That probably is faster than two arrays.

                     

                    Just try importing one BPAV directory into PR and see if that workflow is convenient. File/Import/Directory.

                     

                    Cleaning your Media Cache means new conforming and new PEK file creation.

                     

                    These Xeons are two generations old and no longer considered very fast. Somewhat similar HP systems all score around 450-460 seconds on the PPBM5 Benchmark ,around 4 - 8 times slower than a very fast system, so you can't expect miracles from such a system.

                     

                    I guess that Adobe will not mention number of files and there are people (not me) who claim to edit projects with way over 10,000 clips and complain that CS4 gets sluggish. From those stories, 6,000 clips is a large project but there are even bigger ones. But from a workflow perspective and to keep a semblance of overview, I would cut them up into smaller projects. One for better response and loading times, but also to keep those projects more manageable. The project panel is just not equipped to efficiently handle that number of clips.

                    • 7. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                      hhansard Level 1

                      A xw6800 with dual x5450s scores about 3000 on passmark.  The system is not state-of-the-art, but certainly no slouch (just misses top 20).

                      I would review the drive config.

                      How are the arrays used?

                      Separate assets from program?

                      Separate spindle/array for the scratch/cache files?

                      Separate spindle for pagefile?

                      To improve performance, sometimes it is better to have more/smaller/faster physical spindles and divide your configuration, then fewer larger arrays to store everything in one basket.

                      There are some good posts and Adobe technical docs on system configuration and tuning.

                      • 8. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                        Eric Addison User Group Manager

                        Diane -

                         

                        I shoot with an EX3, and I always import the BPAV folders into PPro with no problems. My usual workflow invovles creating an individual folder for each SxS card I copy over (Card 1, Card 2, etc.), then copying over the entire BPAV folder from the cards into those in folders. Once in PPro, in the import window you should see a button near the bottom right that reads Import Folder - highlight the folder you want to import and then click that button. You'll get an error window saying PPro can't use some of the files - just click past it. Then all your footage should be there and in folders labeled for the folder (and card) number you set up.

                         

                        I can't address the "thousands of clips" issue, but I may have a reason why your having such a rough time. There is an issue with PPro CS5 where XDCAM HD .mxf files that come in separate from the XDCAM HD folder structure take a long time to load in and eat up a lot of memory in the process. Adobe does know about this, but I'm not sure when a fix will be released.  That may be why your seeing such a long load up time.

                         

                        Hope that helps!

                        1 person found this helpful
                        • 9. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                          Harm Millaard Level 7

                          A xw6800 with dual x5450s scores about 3000 on passmark.  The system is not state-of-the-art, but certainly no slouch (just misses top 20).

                           

                          Quite the opposite. It does not get beyond mediocre and you would be lucky to end in the top-200 on that specific test. If you consider that my single i7-920 with only 12 GB scores over 6,055 on that same test, your system can be considered a slouch.

                           

                          Passmark.jpg

                          • 10. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                            Videowilk Level 1

                            I appreciate your "testing my system" Harm, but the point is, it is fully certified (with CS5 and Nvidia CUDA 4800), yet not a new computer - we should all be so lucky. All I am trying to find out is how many clips are too much for Premiere Pro to handle. I am trying to get input from other EX3 users about their workflow - if by using Clip Browser I am somehow making PP create additional unneeded files that slow down loading of the project... my reference to 876 files becoming 37,972. That appears to be my issue, how Premiere handles the clips on import.

                             

                            Yes, we'd all like brand new Z-800s, which I would buy again from my integrator, but the point is about the size of the project and that Adobe says we can work with "thousands of clips in long form projects."

                             

                            I would ask that Adobe list the drivers (and versions of QT) we should be using in order to maximize our workflow. Also, an idea of how to work with "thousands of clips."

                             

                            I appreciate the advice on importing entire folders and will try that with my trimmed down cover footage (reduced from 4,000 to about 1,500).

                             

                            I will also de-select the Write XMP data on import to see if that lessens the number of files created within PP.

                            • 11. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                              hhansard Level 1

                              Sorry for the confusion...  refering to the database of rating scores, not speed scores -- the rating

                              of the Asus P6T w/ 920 is around 1500-2000 with outliers on both ends.

                              • 12. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                Harm Millaard Level 7

                                That is what I'm talking about and I assumed you were talking about as well. Shows what a little tuning can do...

                                 

                                So 3000 versus 6000+ for a dual X5450 versus a single i7-920.

                                • 13. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                  hhansard Level 1

                                  Just reviewed my past 10 projects.  On a quick scan, the expansion from source assets to PP support files was 18:1. Based on your project, it id over 40:1.  Can you break-down what file types are created?  This may help figure-out what is going on.

                                  • 14. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                    Videowilk Level 1

                                    Eric - This is very heplful! I have always used Clip Browser to export mxf for NLE onto my media drives, then import the mxf into Premiere. I had understood that saved a step once in Premiere.Huh. I then use Premiere to view and organize clips into bins.

                                     

                                    It sounds like I should not use Clip Browser at all! Appreciate that advice.

                                     

                                    Do you uncheck the "Write XMP to files on import"? How about "Enable Clip and XMP Metadata linking" ? - in Preferences. Just wondering.

                                     

                                    Thanks,

                                    Diane

                                    • 15. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                      Videowilk Level 1

                                      Hi - Yes, importing the mxf files from the media drive creates pek files - 4 for each clip, klv files and assoc.xml files. As I mentioned before, 876 clips (mxf) turned into 37,972 files.The assoc.xml files seem to be the most in number.

                                       

                                      I do believe I am creating more audio pek files than needed. They always come in on four tracks - and usually I just have audio on one, maybe two tracks. So I'm not sure where my setting is off - in Clip Browser, Premiere or the camera (EX3) itself.

                                       

                                      Eric offerred good advice to NOT use Clip Browser to bring mxf into Premiere because it works better to copy entire BPAV folder in and import the folder (which I didn't know was possible). He indicated that Premiere wants to write extra files without the folder structure present.

                                       

                                      Thanks for all the help,

                                      Diane

                                      • 16. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                        hhansard Level 1

                                        Just scanned my last 10 PP projects.  Did did not find any PP-generated xml files.

                                        • 17. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                          Videowilk Level 1

                                          Do you uncheck the Write xmp files on Import in Preferences hhansard?

                                           

                                          I do believe the culprit is using Clip Browser as I had been... who knew? Obviously, some of you.

                                           

                                          Upon inspection of the media folders, 486 clips (mxf exported from SxS via Clip Browser and into Premiere) this is what was also created, and thus has to load each time Project opens:

                                           

                                          cfa = 1,200 files

                                          klv = 6,553 !!

                                          mxf = 486

                                          pek = 1,944

                                          xdcamhdassoc = a whopping 19,255 !!!!

                                          assoc.xml - 972

                                           

                                          I believe Eric had mentioned that the xdcamhd files are a Clip Browser issue with Premiere.

                                           

                                          Anyone out there know what klv files are?

                                           

                                          Unchecking the Write xmp files on import will do away with the assoc.xml files I presume.

                                           

                                          WOW, I had no idea. Changing my workflow will help me tremendously - thanks Eric and hhansard.

                                          • 18. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                            Eric Addison User Group Manager
                                            function(){return A.apply(null,[this].concat($A(arguments)))}

                                            Videowilk wrote:

                                             

                                            Do you uncheck the "Write XMP to files on import"? How about "Enable Clip and XMP Metadata linking" ? - in Preferences. Just wondering.


                                            I have both checked in my preferences...

                                             

                                            I'm glad to help! Converting to .mxf is an extra step you don't need to do. I very rarely ever use clip browser for anything - not that there's anything wrong with it, just that I don't really ever need it.

                                            • 19. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                              Videowilk Level 1

                                              Thanks Eric - I visited your website too- very nice demo reel!

                                              Cheers,

                                              Diane

                                               

                                               

                                              Diane Wilkins Productions

                                              High Definition Production and Service

                                              850-893-1441

                                              www.dianewilkinsproductions.com

                                              • 20. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                                Eric Addison User Group Manager

                                                Thanks, Diane!

                                                 

                                                Your site and clips look pretty good too - good work!

                                                • 21. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                                  hhansard Level 1

                                                  I have both write XMP ID and XMP Linking checked.

                                                  • 22. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                                    Videowilk Level 1

                                                    Well it's grossly outdated - I badly need to update it!! Thanks anyway..

                                                     

                                                     

                                                    Diane Wilkins Productions

                                                    High Definition Production and Service

                                                    850-893-1441

                                                    www.dianewilkinsproductions.com

                                                    • 23. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                                      hhansard Level 1

                                                      Wondering if your question ever got answered and

                                                      situation resolved?

                                                       

                                                      Did you experiment with the

                                                      write XMP ID and XMP Linking options?

                                                       

                                                      I have a new time lapse project in the offing...interested in your results.

                                                      • 24. Re: So PP CS5 does NOT really allow for thousands of clips, eh? Advice...
                                                        Videowilk Level 1

                                                        Hello- Yes it did. Bypassing the Clip Browser and mxf file import and importing the entire BPAV folder copied from the SxS cards works a great deal better. Apparently Premiere wants to make those pesky xdcamhdassoc, assoc.xml and klv files. Also now only 1 pair of pek files as opposed to the 2 pairs with Clip Browser. I cannot believe how much faster things are now! I have gone back to the default (selecting to do so) on the Write XMP and XMP Linking options - since Eric said he does that.

                                                        Thanks,

                                                        Diane

                                                         

                                                         

                                                        Diane Wilkins Productions

                                                        High Definition Production and Service

                                                        850-893-1441

                                                        www.dianewilkinsproductions.com