15 Replies Latest reply on Jan 11, 2015 8:39 PM by mschubb

    4k Freakout!  Drive speed bottlenecks?

    mschubb Level 1

      Been very happy w/ my 3-year-old CS6 drive setup (thanks to Harm’s recommendations) but I just started shooting a doc in 4K with the Panasonic GH4 and can't believe how fast 4K eats storage.    Have to upgrade my media drives before I transfer media and start the edit and would appreciate help figuring out where to invest to get the best bang for the buck.  Thanks in advance for any discussion, links or advice.

       

      • My rig: Current motherboard (GIGABYTE GA-H57M-USB3) poses one speed bottleneck because it’s only SATA2.
      • CPU is I7-860 w/ 16gb RAM.  Geforce GT240. (planning to get a GTX750ti)
      • Boot drive is 256gb SSD, 128Ggb SSD for exports, plus two internal 7200 1TBs I've been using for HD media.
      • Windows 7 Pro, Adobe CS6 workflow.  (Basic stuff, not a lot in AfterEffects).

       

      Think I’m going to need at least 6TB for working media instead of the current 2gb.  Some questions:

       

      1. Can I just replace my internal media drives w/ 2TBs or 3TBs each, or do you think 7200rpm drives in Sata2 will be a major bottleneck for working with 4K?
      2. How about a USB3 7200rpm external drive for project media?  Is USB3 fast enough?
      3. Would two new internal drives with RAID0 (via OS or motherboard) be a lot better, even in Sata2?
      4. Or would an external RAID be the best way to go?   (Via USB3?  Or eSata?)
      5. OR -- is it time to bite the bullet and get a new motherboard w/ Sata3...?   Or maybe even get a mobo w/ Thunderbolt?

       

      From the many discussions of this topic, I see the best thing would be a new rig.... but can I get by with one of these incremental upgrades and still work efficiently in 4K?

       

      Thanks!

       

      Message was edited by: Mark Schubb

        • 1. Re: 4k Freakout!  Drive speed bottlenecks?
          JFPhoton Level 3

          MS ....Eric Bowen, here on this forum may be best able to advise you on what to do...he is at ADK Computers which specialize in building video editing machines....he knows EXACTLY what you would need to do.

           

          In general, from what I have seen here regarding the minimum requirements for 4k, your current rig is lacking in EVERY area......as well as the software version of PPro you are using.

           

          I read the specs of the footage from your GH4 and see it is a Quicktime format that is MP4 and that runs at almost 50 Mbps. In the past, PPro had huge problems with Quicktime videos.....mostly MOVs.....because Quicktime triggered a 32bit process which CRIPPLED the native 64bit PPro and limited memory use to only 4GB !! I am NOT sure if this is true with the Quicktime MP4 codec from your camera, but, I DO know that the most recent versions of PPro Creative Cloud 2014 have ELIMINATED this issue.....which is STILL PRESENT in previous versions like CS6, 5.5, etc. So, if the "32 bit process trigger" WOULD STILL be present....this ALONE could cause difficulties using CS6 vs. CC2014.

           

          Regarding hardware, YOUR type of 4K is more compressed than other types which have even LARGER file sizes and require large memory on the video card. This means you need a beefy i7 CPU...running at at a high clock speed,( above 4Ghz) with as many cores as you can afford....current Haswell E is great. Then, an Nvidia GPU....a 970 with 4GB DDR5 video memory is good.  SATA III Crucial M550 SSDs are fast and reliable AND cheaper than Samsung 850Pros. The 512 GB version was selling for $160 at the Egg 2 days ago. In your case , 4 tracks of the GH4 footage would only be ~ 24 MB / sec and would not be  a speed bottleneck for those SSDs, or, even quality SATA III HDDs. You COULD assemble a very cheap RAID 0 off the MOBO with the 1TB Seagates for sale at $49 each......four would give you 4 TB of storage running at between 5 and 600 MB/sec. However, you would have to RELIGIOUSLY back that RAID up using an "enterprise level" 4TB HDD, like the Seagate with 128GB cache...its less than $300.

           

          Your current CPU, GPU, and low amount of system memory means you should borrow time on someone's capable machine, or, buy a whole new rig....Eric can advise you the best !....good luck !

          • 2. Re: 4k Freakout!  Drive speed bottlenecks?
            JEShort01 Level 4

            Mark,

             

            4K GH4 + drive upgrades + using i7-860 cpu (launched Q3 2009 at $300 price point) = seems like this might be the right time for a new system!

             


            There are possibly a lot of options - good ones, don't worry - but I would like your answers to several questions first:

            - would you consider a new build (new system) at this time?

            - is your system fast enough? (much faster cpus with lots of cores are available now)

            - would you consider an expensive RAID 5 hardware card that you could use with your current system and later port to a new build?

            - do you currently have a backup plan? And if not, could existing 1TB drives be repurposed into USB 2.0/3.0 sleds?

             

            Regards,

             

            Jim

            • 3. Re: 4k Freakout!  Drive speed bottlenecks?
              mschubb Level 1

              Thanks JFP and Jim.  I understand your points, but I'm really hoping to avoid building a new system for a while...   My "real" work is all 1080P and even this no-budget project shot in 4K will be in a 1080P timeline.   My edit will be a simple rough cut.  If this project goes forward, especially in 4K, we'll need to pass it on to a "real editor" with their own system.

               

              - My system chugs along... fine w/ HD, but some delay when I scrub through 4K clips.  Even HD is slow to render, but I don't render often or a tight deadlines.

              - Was thinking about an external CineRaid box or a GDrive Raid, not a card.  Would an internal card be better?

              - I have a WD 4TB MyBook I use for backup.

               

              The shoot files are not currently in my edit system, but each day gets downloaded to a folder on a 1TB USB3 drive and then a protection copy is made on the 4TB Backup before cards get reformatted.

              • 4. Re: 4k Freakout!  Drive speed bottlenecks?
                JFPhoton Level 3

                ...I saw your MOBO cannot have more than 16GB Ram...that eliminates one simple way of boosting your 4K performance. I don't think 16GB is enough for 4K...even highly compressed 4K. My laptop has 24GB and I see it peaking at 20GB memory usage on some 1080p timelines !! Your GPU is woefully underpowered in trying to tackle 4K....ESPECIALLY if you are going to try and SCALE your work down to 1080p......a task that will run ten times FASTER than "CPU only" using a beefy current NVidia GPU with CUDA acceleration.

                 

                You can easily identify which component is causing the "chugging" by monitoring them......you can see GPU and CPU usage using the free MSI Afterburner utility. You can monitor Ram and disk usage using Windows "task manager". If you see the CPU constantly pegged at 100%, with low ram and GPU usage...you will have your answer. I believe only one track of your GH4 material won't even tax your single drives...unless they are full and fragmented. The stream is only 50Mb/sec...or, 6.25 MB/sec. The bottleneck is more likely your CPU...or, CPU,Ram, AND GPU, if you are trying to add any effects. Try overclocking the CPU using a cooler and viewing a lower resolution in your source and timeline windows.

                Make sure "indexing" is turned off on all your drives. Any spinning HDD should ideally be less than half full and defragmented. Make sure your system is "tuned" and not running unnecessary programs in the background....visit PPBM7 website to see the "tweakers page" and to even run a test on your rig, to identify problem areas in your machine. Make sure your footage is coming from an internal drive, and NOT a slow USB 2 port. Your dilemma reminds me of when I was trying to edit 1080p material on a "core 2 duo" machine......after all, Adobe said that CPU would work right on the box. I wound up transcoding everything and suffering slow, endless and painful sessions trying to complete anything. Thanks to the guys here on this forum, I read about what REALLY worked and was able to extricate myself from the "La Brea Tar pit" of editing with inadequate hardware !!!....................CAN I GET A WITNESS ????........HALLELUJAH !!!!!!....AMEN !!!!!

                • 5. Re: 4k Freakout!  Drive speed bottlenecks?
                  JEShort01 Level 4

                  You answers to my questions helps a lot, thanks.

                   

                  For your situation one of the GRAID models (4/8/12 TB) that have "multi" interfaces (USB 3.0 + Firewire 800 + eSATA) sounds pretty good to me. You would not be gaining redundancy, but until you jump to external devices like the Areca's RAID 5 hardware solutions what is available out there tends to be pretty flaky (better for backups than editing).

                   

                  And for now, use the eSATA connection to the external device with your current motherboard. The early USB 3.0 hardware/firmware/drivers are just not great (your motherboard was one of the first to have USB 3.0). While on paper the eSATA specs may appear slower, you can really count on eSATA for continuous, solid performance. I've not personally used the larger G-RAID model, but I can personally vouch for the G-RAID mini and eSATA.

                   

                  Regards,

                   

                  Jim

                  • 6. Re: 4k Freakout!  Drive speed bottlenecks?
                    ECBowen Most Valuable Participant

                    Working in a 1080 timeline with 4K media doesn't alter the load for that media since the GPU acceleration still has to scale the media down to the 1080 resolution. If you turned GPU acceleration off when you did that and tried playing it back it would bog your CPU down badly. The timeline resolution itself doesn't have the majority of impact on the load for the system and processing. The media and effects does along with any dynamics such as interpolation and audio sample rate change. A raid 0 will help a little but that is really not where your bottleneck is. The video card is not what you want for that media and the scaling your trying to do and the cpu is grossly underpowered for this kind of GPU acceleration. I would off line this project with DNxHD media if you cannot deal with the current performance. Other than that the others are right. You need a new system for 4K media. Even a current Haswell laptop will have more capability than that system.

                     

                    Eric

                    ADK

                    • 7. Re: 4k Freakout!  Drive speed bottlenecks?
                      mschubb Level 1

                      Amen, JFP.  And thanks, JES and JCP.  This weekend I can import some footage and run those suggested diagnostics.

                       

                      It's painful to admit that it's time to spend the $$$ to upgrade the whole system...  (Especially so soon after upgrading the GH2 to a GH4, which led to needing that wonderous 42.5mm Nocticron, which led to wanting to shoot a talking head in 4K (for no damn good reason) for a freebie project, which led needing to several Speed Class U3 64GB cards, etc... All before I noticed that my speedy "new" HD editing box is now 4+ years old.)   Okay I'll stop whining.

                       

                      Seems like two easy upgrades that might get me by til I build a new machine could be :

                       

                      • Spend $160 to swap the old GTX240 (1gb DDR3, 96 cudas) with a GTX 750 TI (2g GDDR5, 640 cudas).  Maybe an Asus Strix which runs especially cool and quiet.
                        • ( I have run the CS hack to make use of the current cuda cores for processing and will run it for the new card.)
                      • Try out an external USB3/eSata RAID0 to hold media.  Maybe the $450 4TB G-RAID w/ two removable drives.
                      • Both items should be useful with the new system I eventually build.
                        • Unless I get a Thunderbolt motherboard; then that eSata G-RAID will have be the bottleneck.  (Unfortunately, the Thunderbolt version of that G-RAID for $550 has ONLY thunderbolt ports and does not have removable drives.)


                      Will know more after this weekend.  Thanks again for everyone's help.


                      BTW, is anyone here using Thunderbolt on a windows machine?

                      • 8. Re: 4k Freakout!  Drive speed bottlenecks?
                        JFPhoton Level 3

                        ... I have a 1.1 Thunderbolt port on my laptop that I use currently to power a 27" WQHD 2560x 1440p IPS monitor. For storage use, Eric has mentioned that Thunderbolt introduces a bit of "latency" compared to other internal interfaces...however, this might be minimal. The newer version of Thunderbolt is 1.2 and supports double the data rate of 1.1......going from around 8Gbits/sec up to 16 Gbits/sec, or, around 2 GB per sec INTERFACE speed. That is FAST !!...for an external port. However, the cost may be HIGH for any storage device which can use that high speed.

                         

                        For NOW, using your current rig, I would visit the PPBM7 website and run the benchmark test....you can see from the individual time results how each component is doing compared to OTHER machines running the same test. The machine may show that it is not worth spending any money to improve.

                         

                        Next, I would try and overclock your CPU up to 4Ghz adding a cheap cooler and test again.......what can you lose ?? You MAY find the strongest NVidia card used for cheap on EBay...you certainly do not want to sink hardly ANY money into this old rig.......it only has TWO SATA II ports and a 16GB memory limit !!!!!  You COULD just buy a 4GB 970 GPU that you can plan to use in a new PC...its $329 at the Egg..

                         

                        Since you only have two SATA II ports, you would HAVE to use Jim's advice and maybe use an eSATA external solution ,for now for the large storage. However, Eric said :   " the cpu is grossly underpowered for this kind of GPU acceleration. I would off line this project with DNxHD media if you cannot deal with the current performance". So, even a new GPU may not do much. Before spending any money I would run the test.....if you overclock the CPU and the CPU portion of the test is not that bad, THEN you may want to try a newer GPU. Then test again before buying any additional storage  to make sure its running OK with the 4K FIRST.

                         

                        In the future, the new Haswell E and X99 motherboard system is LIGHT YEARS ahead of your old rig. It can boot to an M.2 SSD running over 1 GB/sec...or FASTER. Then ,you can just RAID 0 SSDs for your storage on the MANY SATA III ports on the MOBO.  Use at least 32 GB of QUAD CHANNEL MEMORY on it, 64 GB is far better. Throw in a 970,or,980 and do the overclock on an 8 CORE 5960X CPU and yer GOOD TO GO !!!......only 4 to 5 thousand dollars with all the doo-dads !!!

                        • 9. Re: 4k Freakout!  Drive speed bottlenecks?
                          JFPhoton Level 3

                          ....one additional thought. This year, I had to replace my laptop suddenly due to a lightning strike. Lucky me, right then in July I was able to find a powerful Asus i7 G750JW  Haswell gaming laptop REFURBISHED for only $850 , instead of the normal $1,300. It arrived in new condition with the plastic crap still on it. Of course I had to upgrade it. I increased the stock memory from 8 to 24 GB, ( one slot was hidden), or, I would have put 32 GB in. That cost around  $185. Then, the supplied 5400 RPM  HDD was USELESS...but, it has two SATA III HDD internal bays. So, I put two Crucial M500 SSDs in the machine... the M550 BETTER version was just on sale for only $169 each at New Egg, ( the 512GB model).

                           

                          Next, I found a great deal on new Egg for a fantastic 27" WQHD monitor, ( 2560x1440p ) for only $299.....an IPS model from Acer that had  very good technical reviews, ( the K272HUL).....it is AWESOME !!! The pixel density at the 27" size is PHENOMENAL....watching 1440p videos on it is incredible...they are RAZOR sharp with great IPS color....it just blows away the 1080p external monitor I WAS using....it LOOKS like 4K !!! The reason I am mentioning this is because this laptop comes with an HDMI 1.4 port....which allows you to connect a higher resolution WQHD monitor and actually see the full resolution.

                           

                          Now, the best part......that 1.4 HDMI port frees up the Thunderbolt port for a large capacity,high speed external drive......I haven't decided yet what I will get.

                           

                          Although this laptop is no where near the performance of the current Haswell E desktop machines...it IS performing at the level of top desktops from only two years ago !!  the 4700HQ CPU turbos up to 3.25 Ghz on ALL four cores under load and the 765m NVidia GPU runs GREAT with PPro....I even have it overclocked with the free MSI Afterburner utility. When exporting effects heavy ,4 video track 1080p  timelines, I have noticed a 25% reduction in rendering time overclocking the GPU.

                           

                          The current crop of powerful gaming laptops are even MORE powerful....some have the new 4 or 6 GB  970M GPU with a faster Haswell CPU than mine. If working with SIMPLE 4K edits, ( not timelines with many tracks and many effects), you may be able to find a much cheaper 4K solution with a powerful gaming laptop. The newer gaming laptops may have a 1.2 Thunderbolt port, which allows DAISY CHAINING two external monitors, ( 1.1 allows only ONE ). they may ALSO have HDMI 2, for connecting a 4K monitor.

                           

                          I tried to download actual GH4 clips today to try and edit them on my laptop to see if that would help you ,but, I was unsuccessful finding them. So, a powerful gaming laptop COULD be a much cheaper solution than a full blown Haswell E desktop system.....however, you would need to research it very meticulously before deciding. Again, Eric probably already knows exactly how one would behave with your specific workflow....he sells and tests them  at ADK Computers.

                          • 10. Re: 4k Freakout!  Drive speed bottlenecks?
                            cc_merchant Level 4

                            Indicative build suggestions for a Haswell-E system have appeared here: Tweakers Page - Is Haswell-E ready for prime time?

                            Scroll down to the bottom.

                            • 11. Re: 4k Freakout!  Drive speed bottlenecks?
                              mschubb Level 1

                              Trying to run PPBM benchmarks for CS6, but am lost in the weeds.  Website instruction page says there will be detailed instructions in the zip file for "PPBM6 or 7" but the only read me file there offers only this:

                               

                              • If you are testing Premiere Pro CS6, open the PPBM6-DE project file and after finishing the four benchmark steps run the "Statistics PPBM6 CS6-v3.vbs" script.

                               

                              When I open the PPBM6 project file, the on screen instructions say I'm at step two, "MPEG2-DVD test."  What was step one?

                               

                              After steps 2-4, I close Premiere and try to run "Statistics PPBM6 CS6-v3.vbs" but I get a Windows Script Host Error:

                               

                                   Script: E:\PPBM\Statistics PPBM6 Cs6-v3.VBS

                                   Line: 54

                                   Char: 1

                                   Error: File not found

                                   Code: 800A0035

                                   Source: Miscrosoft VBScript runtime error

                               

                              Any suggestions?

                              • 12. Re: 4k Freakout!  Drive speed bottlenecks?
                                JFPhoton Level 3

                                ...the script error is because the exported files are not in the same place as all the "unzipped" files.  many people trip up on this and Bill Gehrke has explained in other posts how to correct it "cutting and pasting" certain files BACK to the proper location.

                                • 13. Re: 4k Freakout!  Drive speed bottlenecks?
                                  JFPhoton Level 3

                                  In addition.....it is best to print out the exact, detailed instructions FIRST, so, you don't miss any steps, or, make mistakes.    You can post your results here on this post, if you would like , so, others may comment on what they reveal. I suspect your CPU may not be that bad....the H.264 test shows that. Your "disk speed" test may be bad if you are using the 7200rpm  HDD. Your GPU acceleration should be weak with your card...I'm guessing.

                                  • 14. Re: 4k Freakout!  Drive speed bottlenecks?
                                    mschubb Level 1

                                    That's my main problem: I did not find a file w/ the exact, detailed instructions included in the Zip file.  

                                     

                                    Searched a bit and didn't find Bill's cut n paste discussion. 

                                     

                                    I think I'm going to download the zip file again and start over....  Maybe download an earlier version; perhaps they just forgot to add the instruction sheet to the latest zip.

                                    • 15. Re: 4k Freakout!  Drive speed bottlenecks?
                                      mschubb Level 1

                                      Thanks.  Got it to work....

                                       

                                      Benchmark says I have a "MPE gain of -14275/115688-0.9"   Here's the data:

                                       

                                      -4937-12918-15688-14275Premiere Version:6.0.5.0

                                       

                                      So what does this result seem to indicate?

                                       

                                      BTW, during the no-CUDA/GPU test, my CPU got REALLY HOT... opened the case and ran a fan on it to keep from going over 80c!  (Other tests stayed under 60c.)

                                       

                                      (my Speccy data is below)

                                       

                                      • Operating System
                                      •     Windows 7 Professional 64-bit SP1
                                      • CPU
                                      •     Intel Core i7 870 @ 2.93GHz    38 °C
                                      •     Lynnfield 45nm Technology
                                      • RAM
                                      •     16.0GB Dual-Channel DDR3 @ 799MHz (10-11-11-28)
                                      • Motherboard
                                      •     Gigabyte Technology Co., Ltd. H57M-USB3 (Socket 1156)    41 °C
                                      • Graphics
                                      •     PA248 (1920x1200@59Hz)
                                      •     ASUS PA238 (1920x1080@60Hz)
                                      •     1024MB NVIDIA GeForce GT 240 (Gigabyte)    40 °C
                                      • Storage
                                      •     111GB ATA INTEL SSDSA2CW12 SCSI Disk Device (SSD)
                                      •     931GB ATA SAMSUNG HD103SJ SCSI Disk Device (SATA)    38 °C
                                      •     931GB ATA SAMSUNG HD103SJ SCSI Disk Device (SATA)    41 °C
                                      •     238GB ATA Samsung SSD 840 SCSI Disk Device (SSD)    29 °C
                                      •     3725GB Western Digital WD My Book 1230 USB Device (USB (SATA))    38 °C
                                      •     1862GB Western Digital WD My Passport 0741 USB Device (USB (SATA))    31 °C
                                      •     279GB Maxtor OneTouch III USB Device (USB)
                                      • Optical Drives
                                      •     ATAPI iHAS424 Y SCSI CdRom Device
                                      • Audio
                                      •     Realtek High Definition Audio