8 Replies Latest reply on Aug 4, 2015 12:32 PM by Generic916

    Adding a RAID card to help speed up export (and other drive question) in Premiere Pro CC

    Generic916

      First of all, I have read Tweakers Page exporting section because that is where my primary concern is. First my questions, then background and my current and proposed configurations:

       

      Question 1: Will adding a hardware RAID controller, such as an LSI MegaRAID remove enough burden from the CPU managing parity on my software RAID 5 that the CPU will jump for joy and export faster?

       

      Question 2: If true to above, then compare thoughts on adding more smaller SSDs for either a one volume RAID 0 or smaller two volume RAID 0 to complement existing HDD RAID 5. That is, I'm thinking of buying four Samsung 850 Pro 128 GB SSDs to put in a four disk volume to handle everything (media/projects, media cache, previews, exports), or split it up into two volumes of two disks each and split the duties, or keep the four disk volume idea and put the previews & exports on my HDD RAID 5 array.

       

      The 850's are rated at SEQ read/write: 550/470 MB/s thus I could get around 2000/1500 MB/s read write in a four disk RAID 0 or 1/2 that if I split into two volumes to minimize volumes from reading/writing at the same time, if that really matters with these SSDs?

       

       

       

      The Tweaker's page made a few comments. One is splitting duties among different disks, rather than a large efficient RAID may actually slow things down. Since the SSDs are much faster than a single HDD, I'm thinking that is no longer accurate, thus I'm leaning toward the Four disk configuration putting OS & Programs on C drive, Media & Projects on D (HDD RAID 5), Pagefile & Media Cache on SSD (2-disk RAID 0) and Previews &Exports on 2nd SSD RAID 0 (or combine the two RAID 0's and their duties).

      Just trying to get a perspective here, since I haven't purchased anything yet. Any experience/stories, I would appreciate.

       

      My current drive configuration:

      My D drive is software RAID 5 consisting of four 1 TB Western Digital RE4 (RED) 7200 RPM HDDs with a CrystalDiskMark SEQ Read/Write of 339/252 MB/s.

      The C drive is SSD 500 GB (Samsung 840 (not Pro) and does 531/330 MB/s. My OS, Program Files and Page File are on C, and data/media files/project, etc all are on the RAID drive.

       

      Problem:

      Current setup allows for smooth editing, only the exporting seems slow, often taking between two and two and a half times the video length to export. Thus a 10 minute video takes 20-30 minutes to export. 15 minute video can take 30-40 minutes to export. The first 10% of the two-pass export takes under a minute (seems fast), but it gets slower where the final 10 or 20% can hang for many minutes like my system is running out of steam. So where is the waste?

       

      I have enabled hardware acceleration (did the GPU hack since my GPU isn't listed) and it may spike at 25% usage a few times and eat up 600 MB of VRAM (I have 2 GB of VRAM), otherwise it is idle the whole export. The CPU may spike at 50% but it doesn't seem overly busy either.

       

      Our timeline is simple with two video streams and two audio streams (a little music and mostly voice) with simple transitions (jump cuts or cross dissolves). We sometimes fast color correct, so that might use the GPU? Also, since we film in 1080 60P and export 1080 29.97 frames/sec, I think that is scaling and uses the GPU. I know without the GPU, it does take a lot longer. I have ruled out buying a faster GPU since it doesn't appear to be breaking a sweat. I just need to know if my system is bottlenecked at the hard drive level because I'm using software RAID and my disks are slow and will hardware RAID significantly reduce the CPU load so it can export faster.

       

      Our files are not huge in nature. Most our clips are several MBs each. Total project files are between 5 GBs and 10 GBs for each video with Windows Media File export being 500 MB to 1.2 GB on average. We shoot using Panasonic camcorders so the original files are AVCHD, I believe (.MTS files?).

       

      Considerations:

      1. I'm thinking of buying (and future proofing) an LSI Logic MegaRAID 9361-8i that is 12Gb/s SAS and SATA (because some current SSDs can exceed the 6Gb/s standard).

      2. I'm not replacing my current RAID 5 HDDs because not in my budget to upgrade to 6 or more large SSDs. These drives are more important to me for temporary storage because I remove the files once backed up. I don't mind a few inexpensive smaller SSDs if they can make a significant difference for editing and exporting.

       

      I can only guess my HDD RAID is slow but the CPU is burdened with parity. I would imagine running RAID 10 would not help much.

       

       

      My setup:

      my setup:

      CPU - i7-3930K CPU @4.5 GHz

      RAM - G.SKILL Ripjaws Z Series 32GB (4 x 8GB) DDR3 2133 @2000

      Motherboard - ASUS P9X79 WS LGA 2011

      GPU - Gigabyte GeForce GTX 660 OC 2GB (performed the compatibility list hack to enable hardware acceleration).

      C drive - 500 GB Samsung 840 SSD (Windows 7 Pro 64 bit and programs).

      D drive - four 1 TB WD RE4 Enterprise HDDs 7200 RPMs in software RAID 5

      Case - Cooler Master HAF X

      CPU Fan - Cooler Master Hyper 212 EVO with 120 mm fan

      Power Supply - Corsair Pro Series AX 850 Watt 80 Plus Gold

      Optical Drive - Pioneer BDR - 208DBK

       

      thanks in advance,

      Eric

        • 1. Re: Adding a RAID card to help speed up export (and other drive question) in Premiere Pro CC
          Generic916 Level 1

          OK, after information overload on everything, I tried (for free), what may be my answer here. I switched codecs on export. When using Windows Media File, Premiere Pro seems to chug along much slower than when using the h.264 (not the h.264 Blu-Ray). I don't know if it's because Microsoft has found a way to break everything or what. But I am getting CPU usage of 70%-85% on the Pass 2 of 2 along with memory usage of 40% (or 12GB). It exported in 14 minutes. woo-hoo. I then did an identical test using Windows Media File. It took 31 minutes to export. The CPU spent much of it's time in the 30's% range, mostly low 30's to mid 40's with spike to 50% and memory usage was only 24%-35%. I have since read how h.264 is the 'preferred' codec for online files. It compresses better with fewer compression artifacts. It looks better, etc. Well, they both looked about the same to me. I'd have to play them both on my 70" TV before I could comment further. In the future, I will consider a RAID controller and upgrading the HDDs to SSDs.

          • 2. Re: Adding a RAID card to help speed up export (and other drive question) in Premiere Pro CC
            JFPhoton Level 3

            ........software RAID 5 off the motherboard ??????......NOT a good idea, from what I have read here on this forum from experts like Harm Millard and others. They have mentioned a LARGE overhead on the CPU doing this....causing sub-par performance. RAID 0 off the motherboard will NOT do this, however.....RAID 0 would provide optimum speed, but, with the risk of total data loss if ANY drive fails. You may wish to reconfigure your RAID to be RAID 0...BUT...you would need to DILIGENTLY back up its entire volume onto perhaps a quality 4TB drive very frequently.

                 A lot depends on the nature of your current and FUTURE codecs you plan to edit. You may not want to sink a lot of money into an older setup that may have trouble with more demanding future codecs. For now, in the 1080p realm, your rig should be OK....the read/write performance on your CURRENT RAID 5 setup is not great, and a definite drag on the performance. The rest of your components appear to be fine.....the Samsung SSD, though not ideal, is OK.....it's write speed is WAY lower than the Pro model,but, the drive is used mainly for reading operations. Since you have Windows 7 Pro, and NOT Windows 8.......you CAN put the entire windows page file onto the RAID 0 you might create.....this will take that frequent read/write load OFF the SSD. Read the "tweakers Page" to see how to best TUNE your machine. To use your current setup most efficiently, without investing much money, you would :a. create the RAID 0 off the motherboard, ( putting all media and project files on it )  b. install a quality 7200rpm 4TB HDD to serve as a BACKUP of the RAID array. Then, install a Crucial M550 256GB or larger SSD, ( close in performance to Samsung 850 Pro...much cheaper), to put all previews, cache , and media cache files on....AND to use as " global performance cache" for After Effects...if you use that program. Exporting can be done to ANOTHER Crucial M550 for best speed...or, just to the either the FIRST Crucial or, the 4TB drive. Your current GPU will accelerate exports on any video containing scaling and any GPU accelerated effects. Your CPU is STILL important in SERVING the data to and from the GPU AND for decoding and encoding non-GPU handled video....your high CPU clock speed helps performance there ! You may want to check out possibly overclocking your video card, using MSI Afterburner.or, similar free program. Increasing the "memory clock speed" can RAISE performance and cut export times on GPU effects loaded timelines,or, scaling operations. On my laptop, I export 25% faster doing this. With my NEW  i7 4700 HQ laptop, I export in the range of your CURRENT machine....about 2 to 3 times the length of the original video. PROPERLY SET UP...your desktop machine should BLOW THIS AWAY !!

                Visit the PPBM7 website and test your current setup to possibly identify current bottlenecks,or, performance issues. THEN, RE-TEST it again, after making improvements to your machine to see how it does. Be aware that new codecs are coming (H.265 and HEVC,etc.) which may demand more computer horsepower to edit, as they are even MORE compressed and engineered for "streaming" high quality at a lower bandwidth on the internet. The new Haswell E...with its quad-channel memory, 8 core option, large number of PCI gen. 3 lanes, goes farther in being prepared for 4K and more. Testing by Eric Bowen has shown the newer PPro versions provide MUCH better processing of 4K than older versions.

            • 3. Re: Adding a RAID card to help speed up export (and other drive question) in Premiere Pro CC
              Generic916 Level 1

              Thank you for your reply. For Christmas, I treated myself and my rig to a LSI MegaRAID 9361-4i and four Samsung 850 PRO 128 GB SSDs. I took out the non-PRO SSD and am using two of the 850s as a C and D drive (using RAID 0 on a boot drive is very tricky), so avoided the RAID 0 and using them separately. I still pointed the usual things to the RAID 5 drives like the Desktop and most of the user profile where the user may want to store files. The rest of the profile resides on the C drive. Programs reside on the D drive. I am using two of the 850 Pros in a RAID 0 for temporary Project files/scratch use. That is, we copy the video files off the cameras to this temp project drive and immediately put a copy on the RAID 5 for safety. After the video is edited, I copy the finished files to the RAID 5 again prior to archiving and delete the files off the temp project drive. The C and D drives plug into the motherboard SATA controller. The other drives all plug into the RAID card. I can now copy 45 GB between the RAID 0 and RAID 5 in about a minute. I can copy videos from the card reader (USB 3) about 4GB/minute on my fast card and 1/2 that on my slower card. The RAID 5 was pathetic before, but it has been reliable. It is now faster, albeit, it is what it is. This RAID card comes with a 1 GB cache, which

               

              Regarding CODECS, I know have more understanding of the pro/cons of exporting to Windows Media File vs. H.264: Windows Media file compresses pretty good on export and is much quicker to upload/process in Youtube, but it's creation takes much longer. H.264 is very fast to export, as fast as 1:1 in terms of video legnth and time to export. However, depending upon the bit rate selected at export, the file may be large or small, with smaller files more subject to issues like pixelation if the subject or camera is moving fast. With larger files, the upload and Youtube processing time is longer. My wife liked the WMF exclusively for a long time. We took a video class at a community college, and she now likes the H.264 format using a large export bit rate.

               

              As a side note, we have upgraded to a Panasonic AG-AC90A at Christmas and just recently took advantage of its XLR audio inputs by getting a Sennheiser wireless mic for it. So we're having a good time filming. We just had such bad workflow process because of the hard drives were slow.

               

              Once again, thanks for responding. I appreciate it.

              • 4. Re: Adding a RAID card to help speed up export (and other drive question) in Premiere Pro CC
                Bill Gehrke Most Valuable Participant

                RAID 5 should never be used for video editing,  It is designed for transactional processing applications   Ideally if not using RAID 0 and its non-redundancy fault you should use RAID 3, 30, 6, or 60.  OF course those are not available on the LSI controller.

                 

                I am using an Areca ARC-1883ix with 7 older Samsung 840 Pro drives in RAID 3 and with a true application benchmark like our PPBM9 and Premiere Pro CC 2015, I am getting ~2000 MB/second write rate performance when exporting from Premiere.  If you register on that site you can see the benchmark results

                 

                If you do not want to go for the ultimate speed with any RAID at all and save a pile of money, just get one of the new Samsung SM951 M.2 PCIe cards and using an Addonics PCIe adapter card, you can get ~1500 MB/second write rate if you have an available PCIe 3.0 x4 or better slot on your motherboard.

                • 5. Re: Adding a RAID card to help speed up export (and other drive question) in Premiere Pro CC
                  Generic916 Level 1

                  Harm pretty much explains why to use RAID 3 as you mentioned, but only that expensive ARECA card supports it. I don't know what the speed difference is in not using it, but that shouldn't apply to me since I export to the RAID 0 SSDs, which are pretty fast.

                  On a side note, I notice Adobe puts lots of files in my Windows profile drive, which I usually clear out once a year (usually several GBs worth). It is good that I'm using a fast SSD for my Windows drive.

                   

                  I had asked my original question because I couldn't believe how slow Premiere Pro exports and wanted to make sure I wasn't doing something stupid, but I had since found out H.264 exports much faster and using faster drives (SSDs) also helps. I realize that my system, while not 'fast' is reasonably balanced and performs decent for what I do. When I make the change to 4K filming, I'll need a new rig since my bus speeds are limited by my CPU (sandy bridge) which would have been nice to know ahead of time. That asterisk on the motherboard saying it was PCIe3 "ready" was a little misleading .

                  • 6. Re: Adding a RAID card to help speed up export (and other drive question) in Premiere Pro CC
                    Generic916 Level 1

                    I just re-read your response. Yeah, I don't have true PCIe 3.0 performance due to the wrong CPU, and my pcie slots are spread out kinda weird, thus the M.2 PCIe cards were not an option for me. I pretty much figure the 850's are giving me 1 GB/sec and the WDs are a fraction of that, but their speed isn't as important right now since all processing is done using the SSDs (C, D, and project drives). But still, the WDs are faster with the dedicated card, so I'm not waiting all day copying between them and the SSDs.   I was not aware you were up to PPBM9. I remember the version 5 and 6 and thought I peeked at 7, but never saw the big results page like the earlier versions. I'll take a look. Thanks.

                    • 7. Re: Adding a RAID card to help speed up export (and other drive question) in Premiere Pro CC
                      ECBowen Most Valuable Participant

                      For future reference, WMV is a very old codec and has not be optimized in a very long time for encoding efficiencies with the current hardware. I would avoid it whenever possible and use H264 instead since the optimizations for that are fresh and continue.

                       

                       

                      Eric

                      ADK

                      • 8. Re: Adding a RAID card to help speed up export (and other drive question) in Premiere Pro CC
                        Generic916 Level 1

                        RE: WMV is a very old codec.....very good point you bring up. It does not export very fast and does not use the CPU threads nearly as good as H.264