26 Replies Latest reply on Apr 18, 2014 11:49 PM by kanegeorgejason

    2k DPX system queries

    kanegeorgejason

      Very soon I'll be receiving footage shot on super 16mm, which is about to be scanned into 2k DPX files (= about 12mb/p frame). I'll be receiving approximately 4tb of footage all up.

       

      I'm searching for some tips on what sort of system I'll need to easily work with this natively.

       

      As I understand, Pp CC handles DPX, so that's where I'm headed software-wise, but it's hardware that I'm uncertain about...

       

      The project won't be effect-heavy, probably just standard grading. All the scenes feature extended takes (no quick back-and-forth edits).

       

      Premiere Pro will be my main concern performance wise - I 'may' use a third party grading software, but that's unclear as of now, so lets stick with Pp performance.

       

      My current system is as follows;

       

      processor: i7-2600 (non-k)

      mobo: p67a-ud5 (only 2 sata3 & 4 sata2)

      Memory: 16GB

      GPU: GTS450

      Disk space thus far: 1tb ssd, 250gb ssd, 1tb hdd

       

      I've got my backup drives sorted too, so my questions only relate to performance, not storage.

       

      My question; what system is suitable for this sort of project? Can i upgrade my current PC to handle it, or should i build a... 4770k? 4930K? Or wait until the release of the haswell-e? If the answer is wait, ill just do an offline cut until the time comes.

       

      Please take into consideration that this project and the next (which will take me through the next few years) will be produced the same way (16mm/2k DPX) so im not looking at creating a monster to 'future proof' me into the 4k-6k-100k resolutions of the future...

       

      And yes, i've read the tweakers page, but i havent built a pc before and need some reassurance/answers for my specific situation.

       

      Many many thanks in advance.

        • 1. Re: 2k DPX system queries
          Bill Gehrke Adobe Community Professional & MVP

          Not knowing anything about DPX, I found this quote with a little searching

           

          "But, the math is simple. file size per frame * fps * 1.2 (little room to spare) per stream you want to play back. That means for 30 fps 2k about 324 mB (note the capital B) per second."

           

          If this is true you will need one humungous RAID system even for one stream.  I have no idea how much CPU power will be required but you are in for a major CPU upgrade.

           

          I hope Eric Bowen comes by this forum thread as he may be able to give you better answers.  It might be wise to call him at ADK

          • 2. Re: 2k DPX system queries
            kanegeorgejason Level 1

            Hi Bill - cheers for responding and a big thanks to you and all the other top contributors, your help is invaluable! Reading these forums and getting up to speed (relatively speaking - I've got plenty more to learn as you can tell!) wouldn't be this easy without this think-tank!

             

            That figure is about right. I didn't explain it thoroughly enough before when I said 12 MB/ per frame - so as I'm working on a 25fps project, it'll roughly be 300 megabytes per second.

             

            As only half of the footage will actually be used in the project during the edit, I was thinking of viewing the footage (in a lower res/compressed format), choosing the best takes, then cutting from there with the full res DPX files (which will effectively be ~2tb).

             

            Now I already own a Samsung evo 1tb SSD, which I know is not a good one to raid with. However, if I had another 1-2tb's of SSD's and split my ~2tb of footage between the ~3tb of SSD's, wouldn't that give me enough speed (in disk terms) without the need of raids?

             

            I'm incredibly new to raids of any kind, so I'm unsure how to go about it. Basically, they scare me and this is why I'm keen on the SSD solution. It also looks like it will cost me an arm and a leg to get a raid with the speeds I need... and with SSD's doing ~500MB read/write, they appear to be the most cost effective way to go for me and this project... but considering my lack of RAID knowledge, I'm sure (or hope) someone can tell me otherwise.

             

            I'm backing up the project outside of my pc (manually, via sata3 HDD's) so any RAID I may use won't need redundancy - just performance.

             

            On the Tweakers Page 'What kind of PC to use', we've got the editable-codecs / cpu-memory-intensity graph. DPX is an image sequence format - so am I right to think it is a part of the 'uncompressed' column?

             

            Again, thanks for the help - I come from a filmmaking background and have never bothered to take the time and understand the hardware I use to edit on - until now!

            • 3. Re: 2k DPX system queries
              ECBowen Most Valuable Participant

              Yes 2 SSD drives in a raid 0 would handle 2K DPX. Otherwise you need to plan on a raid that will handle around 500MB/s in mechanical drives. The Quad Core would not be ideal for DPX even for grading especially if you use Davinci. You will need 32GB of ram.

               

              Eric

              ADK

              1 person found this helpful
              • 4. Re: 2k DPX system queries
                Bill Gehrke Adobe Community Professional & MVP

                Thanks Eric.

                 

                I have two Samsung 840 Pro's that in RAID 0 would easily handle that, they have a transfer rate of over 1000 MB/second, but if you need a couple of TB's online you might have to go with hard drives cost wise..  And as Eric said you really should plan on an upgrade to at least 6 cores.

                 

                I might suggest three Seagate ST2000DM001 drives in RAID 0 if they are on SATA 3 interfaces should be a comfortable solution.  I looked for some of my HDTune similar results but could not find it.  After you setup the RAID array and before you load any data run something like HDTune Read and Write benchmarks to make sure you have the required performance.  Before you setup (load yur OS) a new motherboard make sure you have The BIOS set to RAID for your Intel controller.

                1 person found this helpful
                • 5. Re: 2k DPX system queries
                  kanegeorgejason Level 1

                  Thanks a million for your input Eric.

                   

                  I'll take the 4770k off the table and look at either 4930k or the Haswell-e... It's good to know that something like the 4930k can handle what I'm going to chuck at it, so that's a massive help as I was stuck in theoretical-land without really knowing - cheers!

                   

                  If I go for what's available now - does this sound like a well balanced and able (for my project) pc;

                   

                  • Intel Core i7-4930K 3.4GHz 6-Core
                  • Gigabyte GA-X79-UP4 ATX LGA2011
                  • G.Skill Ripjaws X Series 32GB (4 x 8GB) DDR3-1866
                  • Gigabyte GeForce GTX 770 4GB
                  • 6. Re: 2k DPX system queries
                    kanegeorgejason Level 1

                    Yes, I've had my eye on those Barracuda drives for some sort of RAID0 configuration, however I'm not entirely sure how to do this...

                     

                    Controllers, cards, sas, nas, hardware, software... all words that have me bewildered at my early learning stage... And then there is various published data transfer rates which they all proclaim to be capable of, however consumer reviews seem to say they don't reach these figures...

                     

                    Did I mention that I'm lost? This is the reason why the simplicity of internal sata3 SSD's have appealed to me... however if there's a cheaper option - I'm all ears!

                     

                    For the HDD RAID0 - if I use 4x 2TB Barracudas, which have a 156MB/S read/write speed, am I correct in saying the speeds will approximately be 624MB/S?

                     

                    This is appealing, however, how do I set this up in a cost-effective way? From my understanding you can create a striped volume (RAID0?) just though your motherboard, but on the other hand I've read that this method has its drawbacks... If these drawbacks are too severe, what should I get? A controller? Card? Combo of these things?

                    • 7. Re: 2k DPX system queries
                      ECBowen Most Valuable Participant

                      I would not suggest 4x drives in raid 0. The chance of failure with 1 drive in that raid is to high. That will give you about 450MB/s or so on average. I would suggest looking at more drives and a raid 10 or get a SAS controller and run a raid 6. Keep in mind any raid beyond Raid 1 and you really need enterprise drives or you are likely to deal with drives prematurely dropping out of the raid.


                      Eric

                      ADK

                      1 person found this helpful
                      • 8. Re: 2k DPX system queries
                        kanegeorgejason Level 1

                        Thanks for that warning Eric!

                         

                        When you say more drives on a RAID10 - does that mean internally through the motherboard? Or...?

                         

                        And if I go with the RAID6 via SAS Controller - could you give me an example of the hardware I'd have to buy? Is it something like this? http://www.bhphotovideo.com/c/product/837153-REG/HighPoint_RR2720SGL_Rocket_SAS_6Gb_s_PCI_ Express.html

                         

                        On a side note... if i raid on the mobo internally or not, i have a quick question about the Asrock Extreme9 sata3 connectors... It of course has the x2 Intel ones which are full speed, but what sort of performance can i expect from the 2x Marvell SE9220 & 4x Marvell SE9172? Are they going to give me the performance i need?

                        • 10. Re: 2k DPX system queries
                          kanegeorgejason Level 1

                          Thanks cc_merchant. Do you suggest that one because of performance, or are they different devices?

                           

                          If I were to get the Areca, is it just a case of getting this (below) and connecting each separate internal drive?

                           

                          http://www.bhphotovideo.com/c/product/837184-REG/HighPoint_INT_MS_1M4S_Internal_Mini_SAS_t o_SATA.html

                          • 11. Re: 2k DPX system queries
                            cc_merchant Level 4

                            Look at this one: http://www.bhphotovideo.com/c/product/944300-REG/startech_sas8087s4100_1m_sas_sata_cable.h tml/prm/alsVwDtl

                             

                            It is also called a 'SFF-8087 forward breakout cable'.

                             

                            You get what you pay for. The Areca is the Porsche or Ferrari under the raid controllers, with a fast dedicated IOP and a large fast cache memory. In comparison, the High Point is like a Fiat 500, has no dedicated IOP worth mentioning, is only PCIe-2 capable, has no cache memory and only 8 ports and lacks support for raid3.

                            • 12. Re: 2k DPX system queries
                              ECBowen Most Valuable Participant

                              I second Harm on this. Areca has far better SAS cards than Highpoint especially with performance and rebuild times.

                               

                              Eric

                              ADK

                              • 13. Re: 2k DPX system queries
                                kanegeorgejason Level 1

                                Thanks for giving me the raid 101 cc_merchant, much appreciated!

                                 

                                So if the Areca & Highpoint are at opposite ends of the spectrum - is there any worth mentioning in the middle? Or even the older models of the Areca?

                                • 14. Re: 2k DPX system queries
                                  kanegeorgejason Level 1

                                  Thanks again Eric!

                                   

                                  If i go with an Areca (or similar) raid card, which motherboard would you suggest?

                                   

                                  As far as i can see, the best option would be:

                                  ASRock X79 Extreme9

                                  or

                                  Gigabyte GA-X79-UP4.

                                   

                                  Is the Extreme9 worth the extra $110 over the UP4?

                                  • 15. Re: 2k DPX system queries
                                    ECBowen Most Valuable Participant

                                    Yes it is since that board also supports the E5 Xeon chips as well. That gives you an upgrade path later to the 10 or 12 core chips.

                                     

                                    Eric

                                    ADK

                                    • 16. Re: 2k DPX system queries
                                      Bill Gehrke Adobe Community Professional & MVP

                                      If you really into RAID but want a intermediate level consider the Asrock x79Extreme11.  It has an onboard LSI RAID chip that will give you 8 SATA III ports and it is only $255 more than the Extreme9.  It is limited in that it will not provide the RAID 3 or 6 levels that the Areca does.

                                      • 17. Re: 2k DPX system queries
                                        kanegeorgejason Level 1

                                        Ah, I see. Those Xeon chips are pricey! I don't see myself going down that road anytime soon unless I need to... If this is not the case, is the extreme9 still better value than the UP4?

                                         

                                        And a question from someone who's never built a PC before - if I go with the UP4, will there be enough room in install a GTX760 in the pciex16 slot closest to the ram? (just seems a lot closer to the ram slots compared to the Extreme9)

                                        http://2.bp.blogspot.com/-IFWoHv_lJFA/UHT2vBw_CFI/AAAAAAAAJDg/nh8Qh9wMfgA/s1600/ga-x79-up4 -1.jpg

                                         

                                        Sorry for all the rudimentary questions!

                                        • 18. Re: 2k DPX system queries
                                          kanegeorgejason Level 1

                                          There is a used Extreme11 for sale that can be had for 499USD, so that's a possibility... Is a used mobo advisable? He says its only been used as a show demo for 3 months and works as-new, however he didn't respond when I asked if it came with warranty...

                                           

                                          The 1882i is on sale directly though Areca for $619USD, which is another possibility...

                                           

                                          Would the Extreme11 with onboard LSI in raid10 be noticeably slower than the say UP4 with a 1882i in either raid 10 or 6? (I'll probably run 6-8 hdd's for either setup)

                                           

                                          What would you go for?

                                          • 19. Re: 2k DPX system queries
                                            kanegeorgejason Level 1

                                            Ok, I'm going to get a raid controller but choosing a card seems to be dependant on what raid number I will be going with...

                                             

                                            Eric, you suggested a raid6, is that a better option than raid3? (since i need 300+mb/s)

                                            • 20. Re: 2k DPX system queries
                                              ECBowen Most Valuable Participant

                                              I suggest Raid 6 when possible for the robustness of Raid 6 against corruption.

                                               

                                              Eric

                                              ADK

                                              • 21. Re: 2k DPX system queries
                                                JFPhoton Level 3

                                                ....the upcoming Haswell E  CPU will feature new motherboards that  include TEN SATA III ports....RAID 0 using QUALITY SSDs MAY be one solution, ( i.e. Samsung Pro, or, Crucial M550s ). Bill and Eric would be best to comment on this idea.

                                                • 22. Re: 2k DPX system queries
                                                  kanegeorgejason Level 1

                                                  Thanks again Eric.

                                                  • 23. Re: 2k DPX system queries
                                                    kanegeorgejason Level 1

                                                    Cheers JFPhoton. Yes it would be a solution (and the 10 sata3 ports sounds very inviting) however the SSD route works out to be an expensive venture...

                                                    • 24. Re: 2k DPX system queries
                                                      kanegeorgejason Level 1

                                                      Raid6 it is then!

                                                       

                                                      Last question, promise;

                                                       

                                                      In regards to gpu ram, on the tweakers page it says for HD material go for 1-2GB and for 3K/4K, go 2-4GB... As I'm very close to HD with 2K, will 2GB vram be sufficient? Or with 12MB frames / 300MB/S throughput, will i benefit from a 3/4GB card?

                                                      • 25. Re: 2k DPX system queries
                                                        ECBowen Most Valuable Participant

                                                        If Davinci is in the mix then you really want 3GB. If not then 2GB will do for 2K right now.


                                                        Eric

                                                        ADK

                                                        • 26. Re: 2k DPX system queries
                                                          kanegeorgejason Level 1

                                                          Thanks Eric and everyone else who has contributed to this threat and forum in general - it's helped me immensely.