14 Replies Latest reply on Oct 29, 2013 6:56 AM by RjL190365

    Hard drive performance issue with DPX & MXF files

    Loucann Level 1

      Hi everyone,

      I spent hours on this forum reading articles about hardware to use for editing large files. I also have to admit, that I made some cut backs due to budge limitations.

      I spent most of the money on CPU, motherboard, & ram (6 core  32 gigs , hell of nice MB)  ( Video card= gtx 760 2 gig  )So ok,  I cheaped out on the hard drives. (I think)

       

      I put (2) 7200 rpm  drives (raid 0) as my Operating  system

      (1) External  8TB RAID  G drive (USB 3.0 ) as my Media drive. (footage)

      (2) 7200 drives (Raid 0)  as my Cache Drive

       

      I then opened up an OLD project using canon 5D footage (h264 mov) and some DPX sequence files.

      The 5D fooatge played in real time BUT as soon as it ran into the DPX sequence, the frames dropped.

      I wanted to cry. I just spent 1,600.00 on  parts (&3000.00 pc retail if I bought it instead of building it)

      And I got the same performance as my old quad core 6600/ 6 ram system. WHAT GIVES????

       

      So, can someone recommend the cheapest workaround that can get me editing in realtime?

      Do you think it's the 3.0 usb connection?

       

      The footage I will be editing will be from the canon c300 (MXF files)

      My fear is, if I am dropping frames with DPX file, what's going to happen when I get the MXF files?

       

      Fun Facts:

      I tested my OS drives, cache & media,and I was getting 290-301mb/sec  READ SPEEDS

      Thanks in advance !

        • 1. Re: Hard drive performance issue with DPX & MXF files
          Bill Gehrke Most Valuable Participant

          1.  Most people here would not suggest RAID 0 OS/Application drive.

          2.  You have moderately good read speeds so that should not be a problem.

          3.  Have you tuned this system?   Things like turning off unused applications and processes, turned off disk compression, etc.

          4.  I do not know but there could be a latency issue with USB 3, run a project completely on your internal drive array and see if there is any difference.

           

          Message was edited by: Bill Gehrke

          • 2. Re: Hard drive performance issue with DPX & MXF files
            ECBowen Most Valuable Participant

            Most USB3 raid units do not have a read speed high enough for DPX files. DPX 1080 are roughly 279MB/s. I know the 5 bay Raid 5 USB3 units averaged around 180MB/s with a random access time of 15 to 17 ms. That latency is significantly higher than the internal Sata ports which average 80 to 10ms. I dont expect the 8 bay USb3 raid units to have much better performance. MXF files have a far lower data rate than DPX so if the DSLR H264 files are ok I would not worry  to much about the MXF files.

             

            http://www.hdslr-cinema.com/tools/filesize.php

             

            Eric

            ADK

            • 3. Re: Hard drive performance issue with DPX & MXF files
              Bill Gehrke Most Valuable Participant

              I should be more careful in my reading

               

              "Fun Facts:  I tested my OS drives, cache & media,and I was getting 290-301mb/sec  READ SPEEDS"

               

              Lou:  I hope that your 290-301 mb/secc is really Megbytes per second (MB/sec) and not megabits per second.

               

              And I think Eric meant 8 to 10 ms for internal hard drive latencies. 

               

              Just for future reference here is what two Samsung 840 PRo SSD's can do in RAID 0.when connected to SATA III ports.

               

              Samsung 256 GB x 2 R0 x68 Read 3.png.

              • 4. Re: Hard drive performance issue with DPX & MXF files
                ECBowen Most Valuable Participant

                Yes I meant 8 to 10 ms for the drives on the internal sata controller. Sorry about that.

                 

                Eric

                ADK

                • 5. Re: Hard drive performance issue with DPX & MXF files
                  Loucann Level 1

                  Crap ! Did I make a mistake on the numbers?  I used HD tune pro as well and it said (on first test run)  AVG 301 MB/s acsess time was very high, (13 ? ) I can't remember.

                  also, so the test scores above are 2 solid state drives in RAID 0 ??

                   

                  VERY IMPORTANT:

                  My problem is I have to edit a documentary which contains 6 TBs of clips!

                   

                  How am I going to put all of that DATA on a high performance drive??

                  I would have to take out a 2nd mortgage on my house to buy a 6 TB solid state RAID. 

                   

                  Can you guys PLEASE give me some HDD  options to handle this project?

                  What would you do in my case??

                   

                  (I have around $800.00 bucks) to purchase whatever items I need to edit this film.

                  I already spent $1,600.00 to build the pc, and now I will be at my max at $2,400.00 when I buy the storage space.

                   

                  Thank so much,

                  • 6. Re: Hard drive performance issue with DPX & MXF files
                    ECBowen Most Valuable Participant

                    Can you not break the DPX media up in your workflow to smaller amounts? If not then the cheapest way to get this done would be a 3 drive raid 0 and make sure your project files are backed up daily. I would even look at possible Cloud backup just for a folder where your Project files are located. Project files are not that big so Cloud upload should not take that long. You have the external archive for media with the USb3 storage. If you go with 3 TB drives then you should havethe speed to get this done. 3 Drive raid 0 is not recommended ideally but some times budget outweighs the risk and you just have to micro manage your archives more.


                    Eric

                    ADK

                    • 7. Re: Hard drive performance issue with DPX & MXF files
                      Loucann Level 1

                      So, just to be clear, I should get 3 ( 3TB 7200 rpm) drives = 9TB RAID 0 ??

                       

                      Newegg

                      Seagate Barracuda 7200.14 ST3000DM001

                      3TB 7200 RPM 64MB Cache SATA 6.0Gb/s 3.5"

                      Internal Hard Drive Bare Drive

                       

                      Ok, $120. X 3 = $340.00

                      The company is allowing me  $800.00 for drives

                      so I still have over 400 bucks to spend.

                       

                      So is this the best performance option I have?

                      Thanks so much,

                      • 8. Re: Hard drive performance issue with DPX & MXF files
                        ECBowen Most Valuable Participant

                        That is the best option with the budget you have. A 4 Drive Raid 10 on the onboard would not likely get you the speed you need for DPX 1080 files. A 4 drive raid 5 would require a SAS controller which would exceed your budget. The 3 drive raid 0 would be the only way to get this done with the budget you list.

                         

                        Eric

                        ADK

                        • 9. Re: Hard drive performance issue with DPX & MXF files
                          Loucann Level 1

                          OK, then a 3 RAID 0 it is. Thank you for all your help.

                           

                           

                          also, 

                          "Have you tuned this system?   Things like turning off unused applications and processes, turned off disk compression, etc"

                           

                          No, I did not,.....I never heard of that ?Is that a windows feature?

                          • 10. Re: Hard drive performance issue with DPX & MXF files
                            UlfLaursen Level 2

                            Bill has written this page on tuning Windows 7:

                             

                            http://ppbm5.com/Win7Tuning.html

                             

                             

                             

                            Ulf

                            • 11. Re: Hard drive performance issue with DPX & MXF files
                              Bill Gehrke Most Valuable Participant

                              Loucann,

                               

                              I do not have the data on a three drive RAID 0 but here is a two drive RAID 0 with good Seagate ST2000DM001 drives.  I have to say "good" because there have been two varieties available, some with two platters and some with three platters.  The preferable are with two platters and my experience shows these drive to have an "E" in the third character of the Serial Number like Z1E1ME4V  They are slightly faster than the three platter versions.  Here is the HD Tune Read of the two better ST2000DM001 drives in RAID 0 on my system.

                              HD Tune Seagate 2  x 2 TB Intel Z68 Read.png

                              The best connection is to have all three on a RAIDable Intel SATA III connection but I am doubtfull if this can be accomplished on many motherboards, maybe Eric can comment and enlighten us.

                               

                              Message was edited by: Bill Gehrke

                               

                              What motherboard do you have?  I guess from the info above it is a socket 2011.

                              • 12. Re: Hard drive performance issue with DPX & MXF files
                                ECBowen Most Valuable Participant

                                6Gbs only effects SSD drives stand alone or raided and maybe very large raid arrays. A 3 drive raid 0 will not be effected by that.

                                 

                                Eric

                                ADK

                                • 13. Re: Hard drive performance issue with DPX & MXF files
                                  Loucann Level 1

                                  H again,

                                  Ok, the motherboard is Gigabyte X79-UP4

                                   

                                   

                                  Also you said  " 6Gbs only effects SSD drives stand alone or raided and maybe very large raid arrays. A 3 drive raid 0 will not be effected by that."

                                   

                                  I'm confused. so are you saying that the sata 3g or 6g ports won't make any difference in speed with a 3 raid drive?

                                  If true, then i'm guessing that the 7200 rpm drives aren't fast enough to even reach that 6g speed anyway.??

                                  Is that what you mean?

                                  • 14. Re: Hard drive performance issue with DPX & MXF files
                                    RjL190365 Level 4

                                    Loucann,

                                     

                                    Yes, what Eric stated is that the fastest platter-based (mechanically spinning) hard drives do not utilize anywhere near the full sequential (sustainable) performance potential of even SATA 3.0 Gbps, let alone SATA 6.0 Gbps. The advertised burst transfer speeds of those interfaces are for one channel, not total across all channels. And each SATA channel operates independently of one another as far as transfers are concerned. And when AID 0'd together, the maximum burst transfer rate of AID 0'd SATA 3.0 Gbps drives or drives connected to SATA 3.0 Gbps ports would be roughly 3.0 Gbps multiplied by the number of AID 0'd disks connected to the same controller, which would be just under 9.0 Gbps (theoretical) for a three-disk AID 0 array.