1 2 Previous Next 56 Replies Latest reply on Nov 16, 2012 8:42 PM by Peter Studt

    more than two volumes - real-world benefits?

    Alex - DV411 Level 2

      Anyone finds real-world benefits in creating 3 or 4 different volumes, especially on a laptop?  I see a lot of suggestions on this forum to do exactly that, even on laptops (really?), and find it puzzling.

       

      One thing is shaving seconds off PPBM benchmarks. It's like drag car racing - sure, add that 0.01 psi of pressure, it may help at a slightly higher risk of engine blowing up.  Top fuel cars are rarely practical on a grocery trip though - or on an autocross or oval track.

       

      It's a totally different thing when you tweak the hell out of your system - all for the benfit of saving what, 1 second in 8-hour editing day?  It's money better spent elsewhere, not to mention that multiple volumes also mean multiple points of potential failure.

       

      I usually create two volumes total:

      - (1) OS / boot / apps, and

      - (2) media, which also hosts projects, media cache and all other media-related things.

       

      If (and only if) the client does a lot of faster-than-realtime encoding and exporting, or in some other rather extreme circumstances, there may be another dedicated volume for encoding / exporting output.  Then there may be additional volumes, storage devices for backup and archiving, a whole different topic.

       

      What I am looking for is editing professionals trying both configurations (3-4-tier and 2-tier), and vouching for one or the other, with real-world benefits.  This assumes of course that the editing system is not choking on memory or other things and is otherwise balanced.

       

      Put it another way: outside of people who do a lot of faster-than-realtime encoding, what are the benefits of more than two volumes for editing in Pr?

       

      Thanks!

        • 1. Re: more than two volumes - real-world benefits?
          Harm Millaard Level 7

          Under the assumption we limit this discussion to SATA disks only, the first thing to remember is that SATA is by design a half-duplex connection, in contrast to SAS.

           

          The consequence is that with limited volumes the half-duplex nature will become a bottleneck when a volume is used both for reading and writing of data. Extra physical volumes can avoid that bottleneck by using each volume exclusively for reading only or writing only. That presupposes some knowledge of what file actions are performed by the OS and the application in question.

          • 2. Re: more than two volumes - real-world benefits?
            Alex - DV411 Level 2

            Harm, appreciate the response.  The main question remains unanswered, what are the appreciable real-world benefits of more than two volumes, for Pr purposes.  We can theorize all we want but the reality check remains with real-world professional editors - those who do editing for a living.

             

            (As a side note on half- vs. full-duplex: given that most spinning SATA drives rarely utilize more than 30% of 6G bandwidth, the effects of half-duplex operation are largely negligible in nearly any scenario. Otherwise Nearline SAS drives would be less of a dud than they are.  That said, other benefits of SAS - such as link aggregation and expansion - seem to have much more of a performance impact that full-duplex, in the real world.)

            • 3. Re: more than two volumes - real-world benefits?
              Peter Studt Level 1

              I cannot reply from the perspective of a professional editor but I have some experience and find the topic to be key to my recent research into buying a laptop. Clearly there are no absolute answers to this question, it is more a 'where do you draw the line' type of thing.

               

              Having experienced the frustrations of working with HDDSLR video in CS4 (pre MPE) I know that that experience was below the line of acceptible, it was obstructing my ability to creatively 'flow'. That is not the line you are refering to though, you are talking about going beyond acceptable into 'excess', and the 'end of the day benifit' editors are getting from that.

               

              What is the emotional value that the editing machine has for the user? There is always a quotion of 'owning the cutting edge' excitment, even for the professional who has clients work to attend to, they do not do that work in exclusion of themselves and their own satisfaction for what they are doing. They also try to prevent the 'bottlenecks' that threaten to compromise the extensive investment they have in their editing machines.

               

              Another idea is that, as I describe above, the experience of the frustration from working on a system that is sub-par has created a need to be protected from that, a desire for security of good experience by sticking with the cutting (bleeding) edge of Tech.

               

              The question is deep indeed, and currently the majority (dabblers, new to the field, un-motivated to inquire too deeply) are being influenced by a huge and devious marketing industry. Personally I lean towards your point of view and yet I find myself going to excess in areas, for the reasons above. It does not matter so long as the editors are getting to edit and are not compromising the security of their work and pesonal investments...

               

              I hope that is addressing some of what was on your mind, rather than a facts and figures discussion of benchmarks and the like

               

              Best,

              Peter.

              1 person found this helpful
              • 4. Re: more than two volumes - real-world benefits?
                Alex - DV411 Level 2

                Thanks Peter, this is helpful.  Another thing is that excessive hunt for performance tweaks may adversely affect reliability.  Excessive overclocking is one of them; more than two volumes may be, if, say, two RAID0 volumes are created in place of one RAID5: this would mean two extra failure points potentially bricking your editing machine while the performance gain is questionable.

                • 5. Re: more than two volumes - real-world benefits?
                  Peter Studt Level 1

                  Security is a Big bottom line indeed and a subject with equal depth as the desire for performance. The biggest bottom line must surely be getting the product in the first place, do the creative work, creating the thing needing to be secure...

                   

                  It has to be said that the industry is making many of the choices for us in their struggle to gain market share. CPU's are factory overclocked and tolerances pushed continually. Memory cells in nand chips are internally Raid 0'd and as you say, reliability is the risk verces the extra performance and capacity reward.

                  thanks

                  • 6. Re: more than two volumes - real-world benefits?
                    Harm Millaard Level 7

                    There are complicating issues with the question about number of volumes. It is not so much about the absolute number of volumes, it is more about the sustained transfer rates in combination with the number of volumes.

                     

                    Take two extreme approaches:

                     

                    A. 6 different volumes, a SSD for OS & programs with a sustained transfer rate of 400 MB/s and 5 conventional HDD's with a 150 MB/s transfer rate each.

                     

                    B. 2 different volumes, a SSD for OS & programs with a sustained transfer rate of 400 MB/s and 8 conventional HDD's in raid3/5 with a 750 MB/s transfer rate.

                     

                    This is just an example, and the figures used are imaginary, but the interesting thing in this example is that the total transfer rate of all disks combined are equal. 400 + 5x 150 = 400 + 750.

                     

                    I know that for simplicity sake I would opt for solution B with only two volumes. I also know that in testing a mixed timeline with 7 tracks, including RED 4K and MXF 422 material, I need a sustained transfer rate in excess of 300 MB/s and solution A just does not work, whereas B works fine.

                     

                    Hope this puts your question in more perspective, Alex.

                    • 7. Re: more than two volumes - real-world benefits?
                      Alex - DV411 Level 2

                      I know that for simplicity sake I would opt for solution B with only two volumes. I also know that in testing a mixed timeline with 7 tracks, including RED 4K and MXF 422 material, I need a sustained transfer rate in excess of 300 MB/s and solution A just does not work, whereas B works fine.

                      In other words, you'd choose "B" (two volumes) both for performance and simplicity reasons?  Why are then there are so many recommendations to split them up in three or even four volumes?

                      • 8. Re: more than two volumes - real-world benefits?
                        Peter Studt Level 1

                        Multiple Volumes = more powerful Multi tasking surely.

                         

                        In Harm's example A is broken as far as his requirements are concerned and so the example is not fair, they may be fine working with SD content though. Also there is the added benifit of the extra 3 HDD's in example B where the Raid is true RAID not RAID 0.

                        • 9. Re: more than two volumes - real-world benefits?
                          Alex - DV411 Level 2

                          Multiple Volumes = more powerful Multi tasking surely.

                          Why?  Say, eight drives in (A) four RAID0 volumes, vs. (B) one RAID5 or 6?  Why would (A) offer more powerful multitasking?

                           

                          Even if there were multi-tasking benefits, wouldn't (1) management tasks (some volumes getting full, need to offload to a different volume), (2) lower performance and (3) lack of redundancy far outweigh potential benefits?

                          • 10. Re: more than two volumes - real-world benefits?
                            Harm Millaard Level 7

                            Alex,

                             

                            Your example with 2 disk raid0 versus a large raid5 complicates matters even more, because a 2 disk raid0 suffers from fill rate degradation and a large raid5 does not.

                            • 11. Re: more than two volumes - real-world benefits?
                              Peter Studt Level 1

                              If the volumes in A are large enough and provide sufficient transfer speeds, when compared to one volume they can work on more tasks at once, file management is minimised but still greater, it is a more complex system to set up and run.

                               

                              B will usually offer greater security but at the cost of capacity as always and the multi tasking feature, that others would be better at quantifying than me.

                               

                              The only time A will offer equal security is if the volumes were proper RAID arrays?

                              • 12. Re: more than two volumes - real-world benefits?
                                Alex - DV411 Level 2

                                Your example ... complicates matters even more

                                Let's simplify things a bit then:

                                 

                                Outside of people who do a lot of faster-than-realtime encoding, what are the tangible benefits of more than two volumes for editing in Pr?  (Facts and experiences please, no guessing.)

                                • 13. Re: more than two volumes - real-world benefits?
                                  Scott Chichelli Level 3

                                  quick answer in 2012 absolutely NONE unless you are including an Ext back up..

                                   

                                  all this drive letter nonsense is left over from 5-10 yrs ago when disks were very slow..

                                  rule #1  never read and write from the same drive(s)

                                  rule #2 dont fill past 65%

                                  ideal #1 have a "balanced" system.. (media drive and export shoud be equal)

                                   

                                  past the OS,

                                   

                                  2 sets of 2 drive raid 0 are good enough for most.  this is the most affordable better back up daily

                                  2 sets of 4 drives in raid 0,5 takes care of most things..

                                  1 large 8 drive raid 3,5 is even faster and fast enough to actually read and write from.. (2 sets even better)

                                   

                                  Always even with a raid 6 have a back up, but especally with raid 0..

                                   

                                  rule#3 dont use onboard raid for parity

                                   

                                  Scott

                                  ADK

                                  • 14. Re: more than two volumes - real-world benefits?
                                    Alex - DV411 Level 2

                                    Scott, it doesn't look like you read my question. Seems that you're still advising three volumes (OS, media, export) vs. two, with little if any support from real-world experiences.

                                     

                                    Rule #1 doesn't make any sense if the r/w ratio is highly asymmetric - which in the majority of cases, it is.

                                     

                                    It's nice to see you sharing your knowledge - yet a modicum of a reading effort is appreciated.  Thanks.

                                    • 15. Re: more than two volumes - real-world benefits?
                                      Harm Millaard Level 7

                                      Can't be answered without a clear definition of 'Volume'.

                                       

                                      Is a volume a partition on a single disk or is it a span of multiple disks in a large raid. As long as that is not defined, you can't get a clearcut answer. To illustrate, I'm waiting for the formatting to finish (another 10 hours to go, I think) on a 24 disk array as a single volume. On the other hand I have three volumes on my Win8 boot disk, two of which are system reserved. Do you count those as volumes? Does that mean I have in total 4 volumes on the boot disk plus the array? If so then most people with a single disk in their system have two volumes with a single disk. If they add a disk they end up with at least three volumes.

                                       

                                      You have to be more precise in your question to get a valuable answer.

                                      • 16. Re: more than two volumes - real-world benefits?
                                        Alex - DV411 Level 2

                                        Sorry if it wasn't clear.  By not using a term "partition", I thought it was clear that it's a set of one or more disks visible as a single drive letter to the OS, not a portion of a disk or an existing RAID volume.

                                         

                                        It should also be clear in the context given the rather prevalent advice on this forum to have at least three separate set of disks, or volumes, for OS, media, and output.

                                         

                                        How about now?

                                        • 17. Re: more than two volumes - real-world benefits?
                                          Harm Millaard Level 7

                                          My opinion may not count for much, since I may well be the only idiot on this planet and close vicinity to use such a setup, but here it is, at least for the initial phase of this build:

                                           

                                          1. One disk/volume as boot disk for OS & programs.
                                          2. One disk/volume for Windows Temp files and miscellaneous stuff, like downloads, monitoring software, etc.
                                          3. One disk/volume for stock footage and Sonic Fire audio.
                                          4. One volume for everything video related, media, projects, media cache, previews and exports.

                                           

                                          What are the 'tangible' benefits of such an approach?

                                           

                                          1. Simple installation of programs, no worries about Users/<username>/AppData files, etcetera.
                                          2. Reduces the writes on the C: SSD and improves longevity, while reducing the half-duplex bottleneck by using a 2-nd SSD.
                                          3. Not used very often, but I want them stored locally, not on the server to improve performance when needed.
                                          4. Easy administration, just one volume for everything video related. But this is a 3 x (7 R3) plus 3 global hot-spares raid30 array. Once it is finished formatting I can start testing this setup, but I expect a sustained transfer rate in excess of 1,000 - 1,250 MB/s on this volume and have safety for multiple disk failures at the same time. Six disks are parity disks so I only have 18 TB net space, but that is the cost of safety.

                                           

                                          I know, I'm not the 'Average Joe' when it comes to PC's, but I also have two servers for off-loading stuff I don't need on a daily basis or that is not performance critical that is stored on one of the servers and backups to the other server over VPN with a NAS doing incremental backups too, helped by a UPS and using teamed Gb NIC's.

                                           

                                          Since this is only the start of this build, I plan to have - somewhere in the future, funds allowing - 2 additional SSD's in raid0 as a separate volume to maybe reduce the load on the main array for media cache and previews. Additionally I may change the single disk/volume, now used for stock footage and Sonic Fire audio to a 5 disk R3 array, one volume, and put exports there.

                                           

                                          Administratively this is all pretty easy and ensures that all (raided) volumes have great transfer rates, giving a very snappy feeling to the editing, but at the same time giving security in case of disk failure.

                                           

                                          Sorry I can't give you straight figures like standardized fuel consumption on cars, because there are too many variables involved for meaningful figures.

                                          • 18. Re: more than two volumes - real-world benefits?
                                            Alex - DV411 Level 2

                                            Harm, you are certainly not an idiot, and your setup has a lot going for it.  In fact, it has two main volumes: OS, media, and no dedicated export / output / encoding one.  The other volumes aren't critical or hugely beneficial for editing purposes.

                                             

                                            Another way to phrase my original question:

                                             

                                            "For a balanced and well performing system configured for Adobe CS Production Premium apps, is it recommended to have more than two volumes?"  If so, what are the tangible benefits?  E.g. how many hours, minutes or seconds will three+ volumes save in time for an average pro editor, on any given workday?

                                             

                                            Based on my 17 years experience configuring editing systems, and feedback I am getting so far, I suspect that the answer is "no (real-world benefits)" and in fact it's not recommended to split the media volume in two or more.

                                             

                                            While your system is more "extreme" rather than "balanced", it validates the idea that two volumes are sufficient for most editing purposes; more than two volumes should only be recommended in extreme circumstances.  Would you agree?

                                            • 19. Re: more than two volumes - real-world benefits?
                                              Peter Studt Level 1

                                              Hi Harm,

                                               

                                              Would you say then:

                                              1/ Reduction of excess wear from writing to the SSD would be one benifit.

                                              2/ Assisting file management by having individul volumes for different forms of application.

                                              3/ Having different volumes for different aspects of the primary purpose (video editing), or what I was calling 'Multi tasking.

                                               

                                              Did I miss anything?

                                               

                                              A quick question for someone. Occasionally after rendering (Vegas was plagued by this in my experience) there are 'glitches' in the video, similarly working with audio there may be pops and the like, would this be minimised by having an exclusive controller working on the export? Or would these originate more in the Processor dept, CPU GPU?

                                              • 20. Re: more than two volumes - real-world benefits?
                                                Harm Millaard Level 7

                                                Alex,

                                                 

                                                I still think it is more a question of sustained transfer rates than number of volumes that dictate what is ideal in an individual situation. In my specific case, where I invested around 17% of the total system cost in a dedicated raid controller, I have sufficient transfer rate capability to make life easy for me by using effectively a single volume for video material. But not everybody has that luxury or is willing to spend the $$$. So, the bottom line is, it depends.

                                                 

                                                Peter,

                                                 

                                                1/ It was indeed the reason to spread the reads (mostly from C:) and some writes (mainly to D:). To be honest, I have been contemplating moving my user profile to D: as well, but left that decision for a later moment. Notice however that the My Documents folder and My Music, Pictures, Video etc. are all located on a server, not locally.

                                                2/ Not really, it was more a practical decision to make administration easy and do not forget, this build is only for video editing. I still have a laptop for Office work and I have 'Harm's  Beast' (with all its 17 disks) for website development, internet and other stuff. (You should see my desk, 5 monitors, 4 keyboards, 2 printers, 2 mice, a Contour Shuttle, and a lot of other paraphernalia).

                                                3/ This I don't understand. What do you mean?

                                                 

                                                To reply to your quick question, well, it is not a reply but merely a guess at this stage, but hiccups like that are often caused by a bottleneck in the system when OS, background processes, services and the application running are all demanding resources, but only one gets them and the rest has to wait or make do with the limited resources available at that moment. In severe cases this often causes the dreaded 'white screen' or 'not responding' symptoms. As to how to locate these bottlenecks, that is a different story. It requires an overview of the processes and services running, complete hardware info and the application, but a good way to start troubleshooting these kind of problems is looking closely at both DXDIAG results and Process Explorer lists. Often these can help a long way.

                                                 

                                                PS. Initial test on this single volume gives me around 1281 MB/s reads, and (with 4 GB cache) around 2628 MB/s writes.

                                                 

                                                Areca_E.png

                                                Areca transfer rartes.png

                                                • 21. Re: more than two volumes - real-world benefits?
                                                  Peter Studt Level 1

                                                  As your system stands right now (after the format), ignoring your OS type volumes, is a 1 volume editing system, (assuming stock footage is not related to editing?).

                                                   

                                                  But you then say

                                                  Since this is only the start of this build, I plan to have - somewhere in the future, funds allowing - 2 additional SSD's in raid0 as a separate volume to maybe reduce the load on the main array for media cache and previews. Additionally I may change the single disk/volume, now used for stock footage and Sonic Fire audio to a 5 disk R3 array, one volume, and put exports there.

                                                   

                                                  Which means you plan to go from a 1 volume system to a 3 volume system.

                                                  Perhaps if you difined "Reduce the load" it would point us to your exact rationale for using a multi volume setup over a single volume one?

                                                   

                                                  I have always assumed putting project files, media files, media caches, export, on seperate volumes was to allow PPro to concurrently access different data. Rather than having one controller doing it all on a single volume you can then have a controller for each aspect of PPro's data requirements, reading/writting cache files, reading media files, writing exports, writing project files.

                                                  I believe I have arrived at this assumption from reading reccomendations to use multiple volumes to boost PPro's performance (not multiple Drives in RAID but multiple volumes as we have defined them in this discussion here).

                                                   

                                                  Multi Tasking for a data storage system = concurrent access, write or read, to different data sets.

                                                   

                                                  The only other reason I was wondering about was the possible benifits from having uninterupted writing of exports thanks to the exclusive controller. Perhaps this is not an issue today but from the past? As I mention before my experience of Exporting with Sony Vegas was fraught with 'glitches' but PPro on the same machine 99.99% cured the glitches.

                                                   

                                                  I fully accept my assumptions are not based on a comprehensive knowledge, more like a Drissel of knowledge!

                                                  Hope that helps

                                                  • 22. Re: more than two volumes - real-world benefits?
                                                    Alex - DV411 Level 2

                                                    I still think it is more a question of sustained transfer rates than number of volumes

                                                    Would we be able to produce a specific example highlighting the need for more than two volumes?

                                                     

                                                    E.g. if you only had two disks to work with (besides OS/boot), what's the benefit of making them separate rather than bundling into RAID1 or 0 for media?  Seemingly, the latter is better for transfer rates, as well?

                                                    • 23. Re: more than two volumes - real-world benefits?
                                                      Peter Studt Level 1

                                                      Hi Alex.

                                                       

                                                      Lastnight I found a Table that Harm created that supports your thoughts, Guidelines for disk usage. Here you see recommendations for separate volumes until you reach 5 disk, where RAID appears and it becomes a 2 volume setup again.

                                                       

                                                      What I am learning is that performance and data security are on sliding scales, being modified by Cost and Capacity. Further complicated by File and Application managment and considerations for SSD lifespan.

                                                       

                                                      I think it would help if Harm could help us further by replying to my last question to him, asking him to define or clarify why he is planning to go from a 'simple' 2 volume setup (OS and Editing volumes) to a 4 volume setup in the future. Clearly he has the performance slider and the security slider set to max!

                                                       

                                                      Note to Harm: your setup is totally mindblowing!!! I liked hearing the description of your computer desk with all the keyboards and monitors and stuff, it would be intrigueing to see it

                                                      • 24. Re: more than two volumes - real-world benefits?
                                                        Harm Millaard Level 7

                                                        Alex,

                                                         

                                                        E.g. if you only had two disks to work with (besides OS/boot), what's the benefit of making them separate rather than bundling into RAID1 or 0 for media?  Seemingly, the latter is better for transfer rates, as well?

                                                         

                                                        If you have two disks of 1 TB each to work with and let's say they are modern high speed disks with a sustained transfer rate of around 150 MB/s each, these are the options you have:

                                                         

                                                        1. Two separate volumes, 150 MB/s transfer rate each and 2 TB capacity in total.
                                                        2. One volume in raid0, 300 MB/s transfer rate and 2 TB capacity in total.
                                                        3. One volume in raid1, 150 MB/s transfer rate total (the other transfer rate is usurped by the mirror) and 1 TB capacity.

                                                         

                                                        Looking at these options, solution 2 makes it easy to administer, gives you around twice the speed of a single disk/volume, but also doubles the risk of data loss. Solution 1 requires some knowledge of how the OS & application work to distribute the load evenly in order to get around the same performance as solution 2, but does not double the risk of data loss as solution 2 does. Solution 3 is for people that don't mind the extra cost of an extra disk but buy security without performance benefits, probably because they hate restoring backups.

                                                         

                                                        Peter, let me get into an analogy to clear up what I mean with "Reduce the load".

                                                         

                                                        I have now a super fast raid array with a sustained transfer rate of around 1,300 MB/s, similar to a 6-lane highway, but even that highway can get struck with traffic jams in rush hour. If you have a serious traffic jam (maybe an overturned truck or a massive collision) it may be worth to take an alternative route.

                                                         

                                                        The PCIe-3.0 bus has to accommodate all the traffic over this (raid) highway, which theoretically can accommodate enough traffic without jams, but in some cases weather conditions or number of cars, incidents, etcetera can still cause traffic jams. For those moments I consider using a parallel lane that is still unused (2 SSD slots on the Marvell controller) or - if the government is going to increase the highway to 8 lanes - to use the remaining 4 ports on my controller.

                                                         

                                                        I will post some pictures of my desk later.

                                                        • 25. Re: more than two volumes - real-world benefits?
                                                          Scott Chichelli Level 3

                                                          we intend to benchmark this again, we had in the past but its too old (CS3 i think)  Eric and i have also been writing docs for a "why" section (for months)  here is one snippet the latter part being important..

                                                           

                                                          until the benchmarks take it or leave it,  the heavier the codec the more obvious this becomes.. try working with red4k on a single volume and play back @ even 1/2 screen with multiple layers/effects.. good luck unless its a large raid array like harms..

                                                           

                                                          The final point of focus is the hard disks. Hard Disk load is inherent to both centralized processing and GPU acceleration. However the effect on GPU acceleration is greater due to the far more time spent playing the timeline back in real time versus a preview file rendered to disk. When a rendered timeline is played then the disk load is decided by the bit rate of the preview codec. This is because the rendered timeline is simply a preview file created on a disk. The bit rate of the preview codec often changes depending on the project type. However that does make it easier to decide the overall disk requirements. The Realtime playback however is playing back multiple files. The advanced player is reading each media file for decoding processing. Each of these media files have their own bit rate requirements. Along with this cache files are also read in order that the Realtime playback can process all of the changes made by the editor. This of course exponentially increases the bit rate load on your disks. Now the editor has to account for the bit rates of each media type in the timeline. Add to this that mechanical drives have a significant latency between each read and write request. This is why codecs with even smaller bit rates like AVCHD still perform better with raid arrays. Instead of 1 drive handling all of the read requests by a timeline, now you have multiple drives handling the same amount of requests. Since video requires a specific amount of data to process in 1 second of time to play back cleanly, this is critical to the Realtime playback performance and thus the editor's experience. The simple focus point here is exceed the bit rate of the lowest compression codec used by several times. Then expect multiple disks for every set of 4 to 5 tracks or so to handle the requests. The Render/export times are also effected by the disks because now you have multiple read and write requests to assign. This is why it's better to split the read and write load between 2 disks or volumes depending on whether you require smaller or larger raid arrays.

                                                           

                                                          Scott

                                                          ADK

                                                          • 26. Re: more than two volumes - real-world benefits?
                                                            Peter Studt Level 1

                                                            Hi Scott, that is very clearly put, I thank you.

                                                             

                                                            It tells me that paramount in Video Editing is performance (transfer rate), the more excess you have the more sophisticated and complex the project can be. Cost then dictates the use of RAID 0 for most as a way of providing the transfer speed, others take a more complete route by providing data security and performance.

                                                             

                                                            Next it confirms the multi tasking nature of PPro, the timeline contains references to data contained in multiple folders, 3 user data folders; 1/ video from the users media 2/ audio from the cached conversions made by PPro 3/ preview files. It needs access to up to two locations simultaneously. During export it needs access to those two locations at the same as writing to a 4th location. Its random access in nature.

                                                             

                                                            Having said that though, it is really only important for the writing, so long as you have enough transfer rate. More volumes being serviced by their own controller and data channels provides more transfer rate, plus a miniscule bonus from simultaneous access.

                                                             

                                                            Working with increasingly huge resolutions (the little GoPro Hero3 records 4K @ 15fps as an example, it won't be long before DSLR's might be working in 4 even 5k) our ideas of excessive become required! Harms large array begins to look less like a Rocket ship that can't even use the highway and more like a high performance jaguar racing car (I suspect Harm would like the jag), alittle out of place around town but on the highway it owns 4K!

                                                             

                                                            If semi's are going to be causing holdups, they should be banned from the highways! just joking

                                                            • 27. Re: more than two volumes - real-world benefits?
                                                              Harm Millaard Level 7

                                                              The Jag XKR is a nice car, but so is the F12 berlinetta. However, for everyday use the Jag may be the better choice, but then again the Porsche Cayenne Turbo S may be nicer for the family.

                                                              • 28. Re: more than two volumes - real-world benefits?
                                                                Peter Studt Level 1

                                                                LOL Harm

                                                                I like the Mercedes E55 AMG, but I'm sure if I had the chance to test drive some others I might find others I would like aswell.

                                                                 

                                                                F12 berlinetta is Beautiful!!!

                                                                • 29. Re: more than two volumes - real-world benefits?
                                                                  Alex - DV411 Level 2

                                                                  try working with red4k on a single volume and play back @ even 1/2 screen with multiple layers/effects..

                                                                  Surely you meant "disk"?  A "volume" can be anything from a fraction of a single disk (like Harm said) to an array consisting of thousands of disks where transfer rates become meaningless (bottlenecks are in connectivity and OS/FS overheads) and it's all about latencies and TPS.

                                                                   

                                                                  While a general recommendation for more than two volumes on any given budget is puzzling to me (I've explained above, why), even more puzzling is this tendency to steer away from the topic.  E.g. "can't do multiple 4K layers from a single disk".  Who said anything about a single disk?  Or, "If it's a partition, it's not gonna work too well."  Sure, let's also include network shares, thin provisioning, iSCSI latencies and how expensive metal is in China.  But why?

                                                                   

                                                                  The question couldn't have been simpler: "what's the most efficient and logical disk configuration for editing in terms of number volumes?"  Did we really need to touch on partitioning or how 4K and DPX files won't work well off of a single disk?

                                                                   

                                                                  Regardless, let's simplify things again.  My thesis is this:

                                                                   

                                                                  When the volumes have sufficient performance for the task, there is rarely a real need for more than two volumes.

                                                                   

                                                                  Another angle at it is this:

                                                                   

                                                                  For any number of available disks, or available budget, the most efficient configuration in the majority of cases, is two volumes: (1) boot/OS/apps/othercrap, and (2) media and everything media-related.

                                                                   

                                                                  Any objections or do we need to simplify things even further?

                                                                  • 30. Re: more than two volumes - real-world benefits?
                                                                    Peter Studt Level 1

                                                                    Alex wrote:

                                                                    For any number of available disks, or available budget, the most efficient configuration in the majority of cases, is two volumes: (1) boot/OS/apps/othercrap, and (2) media and everything media-related.

                                                                    Define 'efficient'?

                                                                     

                                                                    2 disks: simple

                                                                    3 disks: I would say make 3 volumes so that a separate volume is available to write exports.

                                                                    4 or more disks: it becomes a trade off of transfer rate vs data security vs practicality.

                                                                     

                                                                    Is it efficient to have work done being at risk of loss? Also is it efficient to create one volume from a bunch of missmatched disks?

                                                                     

                                                                    From my new comprehension of this, I would say in ideal circumstances I would go for a 3 volume setup, adding one volume to your 2 volumes so that exports can have a volume to themselves. Anymore is perhaps not ideal efficiency but efficiency does not exist in a vacuum, there are other practicalities that mitigate complexity in a system.

                                                                     

                                                                    Even a setup as highly planned and evolved as Harm's one does not just appear with a wave of a wand, there are cost and availability to consider, different types of media (HDD vs SSD), practicality of modifying an existing volume.

                                                                     

                                                                    I apologise if I seem to be pipeing up excessively, but I feel putting myself in this discussion at this level has been of considerable benifit to my understanding, and by trying to put things into plain english might have helped the comprehension of highly technical concepts that many reading may find obscure. I think this is my last post to this thread. Thanks to all, very much, it has been enlightening and great value! Thanks  to Alex for the incissive and persistant inquiry.

                                                                     

                                                                    Laters

                                                                    • 31. Re: more than two volumes - real-world benefits?
                                                                      Alex - DV411 Level 2

                                                                      Peter,

                                                                       

                                                                      "Efficient" means the easiest and simplest way (in terms of management and maintenance overhead) to achieve the desired result.

                                                                       

                                                                      From Google: Adjective: (esp. of a system or machine) "Achieving maximum productivity with minimum wasted effort or expense."

                                                                       

                                                                      I am still not sure why we're talking about disks and transfer rates if the thesis says, "When the volumes have sufficient performance for the task".

                                                                       

                                                                      In other words, if you'd like to question the thesis, please come up with a common example where such configuration would be undesirable or less efficient vs. some other configuration.

                                                                       

                                                                      E.g. where a dedicated export / output volume is concerned, what are the benefits of doing that vs. adding the disk(s) from this volume to the main media volume and thus increasing the performance of the main volume that is used much more frequently and has higher performance demands?  If you believe there are such benefits, let's detail the configuration and substantiate the benefits based on the same disks.  A common example:

                                                                       

                                                                      We have three disks: all 2TB 7200rpm SATA 6G.

                                                                       

                                                                      Configuration 1 (two volumes):

                                                                      1. OS, apps, etc. - one 2TB disk (transfer rates: 70-140MB/s)
                                                                      2. Media - two 2TB disks in RAID0  (transfer rates: 130-260MB/s)

                                                                       

                                                                      Configuration 2 (three volumes):

                                                                      1. OS, apps, etc. - one 2TB disk (same as above)
                                                                      2. Media - one 2TB disk
                                                                      3. Export / Output - one 2TB disk

                                                                       

                                                                      Which is a more logical and efficient configuration for most editing workflows and why?

                                                                       

                                                                      (Please note: common scenarios, not extreme.  Most editors do not export / output all day long, and do that once or twice per editing session at best, and even then - the output is nearly always a highly compressed single stream, and often slower-than-realtime making a dedicated export volume an unnecessary excess, complexity, and a potential point of failure - which was already discussed above.)

                                                                      • 32. Re: more than two volumes - real-world benefits?
                                                                        Alex - DV411 Level 2

                                                                        Folks, you'll have to excuse me for being so stubborn and persistent. 

                                                                         

                                                                        Personally, I believe that the benefits of having a bunch of different volumes dedicated to media cache, swap files, and export - while making sense for some scenarios, do not make a lot of sense in most common workflows.  I'd like set it straight, or set myself straight.    Whatever consensus we achieve, I'll be sending people to this thread so they can figure out for themselves, what is the best scenario / configuration for them and their workflow.

                                                                         

                                                                        This is why I'd like to kindly ask you to pay attention to the "ifs" and "whens" of this thread:

                                                                        • we assume the performance is already sufficient
                                                                        • same number / type of disks in all configurations
                                                                        • common (and not extreme) scenarios

                                                                         

                                                                        Thank you.

                                                                        • 33. Re: more than two volumes - real-world benefits?
                                                                          Alex - DV411 Level 2

                                                                          If you have two disks of 1 TB each to work with and let's say they are modern high speed disks with a sustained transfer rate of around 150 MB/s each, these are the options you have:

                                                                           

                                                                          1. Two separate volumes, 150 MB/s transfer rate each and 2 TB capacity in total.
                                                                          2. One volume in raid0, 300 MB/s transfer rate and 2 TB capacity in total.
                                                                          3. One volume in raid1, 150 MB/s transfer rate total (the other transfer rate is usurped by the mirror) and 1 TB capacity.

                                                                           

                                                                          Looking at these options, solution 2 makes it easy to administer, gives you around twice the speed of a single disk/volume, but also doubles the risk of data loss. Solution 1 requires some knowledge of how the OS & application work to distribute the load evenly in order to get around the same performance as solution 2, but does not double the risk of data loss as solution 2 does. Solution 3 is for people that don't mind the extra cost of an extra disk but buy security without performance benefits, probably because they hate restoring backups.

                                                                          Probably worth mentioning that mirroring is not backup.  Data wiped from a mirror via file system corruption or user error is gone.

                                                                           

                                                                          Also, as discussed above, let's assume backup is taken care of - via network shares, external drives, etc.

                                                                           

                                                                          Then if we take into account efficiency and ease of configuration and use, then separate volumes (1) make the least sense, RAID0 (2) - the most, and mirror (3) - only if it's fast enough and if redundancy is more important than speed and capacity.

                                                                           

                                                                          Seems like a no-brainer?

                                                                          • 34. Re: more than two volumes - real-world benefits?
                                                                            Alex - DV411 Level 2

                                                                            Lastnight I found a Table that Harm created that supports your thoughts, Guidelines for disk usage. Here you see recommendations for separate volumes until you reach 5 disk, where RAID appears and it becomes a 2 volume setup again.

                                                                            I respect Harm and am 200% positive a lot of thought went into this table.  I hope he doesn't think of it as set in stone and good for all cases without exceptions, though.

                                                                             

                                                                            Personally I disagree with it where it advises more than two volumes - since it's a generic recommendation not based on specific workflows.  I also see lots of other folks offering similar recommendations, like, "use dedicated volumes for export and media cache".  This does not make any sense to me.  Those dedicated volumes, in 90% cases, are better used as part of the media volume - great if with redundancy - with routine back up if not.

                                                                             

                                                                            I can only guess PPBM5 scores had some influence on that - which only affirms the idea that the advice (of more than two volumes) is good mostly for extreme scenarios, since PPBM5 is hardly representative of common editing workflows.

                                                                            • 35. Re: more than two volumes - real-world benefits?
                                                                              Peter Studt Level 1

                                                                              Haha, I should not have said my farewells in my last post, because this discussion is by no means a wasted effort and clearly is not resolved to the point of 'mutual' satisfaction'. And it takes my mind of off my aching tooth...

                                                                               

                                                                              A question for you Alex; since you argue for simplicity, would you advocate a system of 1 single volume, for everything? No separate OS/Applications drive, just one large and powerful volume to do everything?

                                                                               

                                                                              For me the answer to this question is a 'no brainer' firstly because of the existence of SSD's but more on the topic, because OS and Apps are so clearly (OS at least) processes that are unrelated to the editing processes. BUT IS this another 'Generic Recommendation'?? Are there clearly and easily recognised benifits to even having more than one volume in the whole computer? Is the whole idea of separating disk access processes just a concept that seemed like a good idea back in the day and now is seen as obvious wisdom without any ongoing questioning?

                                                                               

                                                                              This is taking your proposition to the Nth degree and I would be interested to hear your and others views.

                                                                              • 36. Re: more than two volumes - real-world benefits?
                                                                                Alex - DV411 Level 2

                                                                                A question for you Alex; since you argue for simplicity, would you advocate a system of 1 single volume, for everything? No separate OS/Applications drive, just one large and powerful volume to do everything?

                                                                                I wouldn't - because (1) both are essential to the system (unlike export, media cache, swap file, etc.) and (2) each have their own distinct performance and configuration requirements.  None of this applies to any other volume type: none of them are critical, at least not commonly.

                                                                                 

                                                                                That said, it does depend on the usage (workflow).  If I only edit once a week for a few hours, and the system drive is fast and large enough, why the hell not? (None of my customers fall into that category though.)

                                                                                 

                                                                                Priorities for various volume types:

                                                                                1. boot/OS/apps: important: low latencies.  Less important: capacity, bandwidth (compared to media).
                                                                                2. Media: capacity, bandwidth (sustained transfer rates), possibly redundancy.  Less important: low latencies.
                                                                                3. export: no specific requirements. Nearly any drive is fast enough and large enough for, say, YourTube H.264 files.  Extreme requirements such as 4K 16-bit DPX files, may change that of course.
                                                                                4. media cache: transfer rates.
                                                                                5. swap file: low latencies, possibly transfer rates. However if your swap file starts getting pegged too often, it's time to re-tune or upgrade your system.

                                                                                 

                                                                                From all this, it's pretty clear that the two essential volumes (OS, Media) have requirements distinct enough for them to be separated physically.  The rest? They are not essential, and other volumes (either media or OS) can usually easily perform their functions.

                                                                                 

                                                                                Which begs the question: why dedicate separate physical disks to non-essential volumes when they can instead be used to increase performance, capacity and/or resiliency of the essential ones?

                                                                                 

                                                                                (Thank you Peter, your question helped me articulate my thoughts more clearly, at least for myself. )

                                                                                • 37. Re: more than two volumes - real-world benefits?
                                                                                  Peter Studt Level 1

                                                                                  Thanks Alex, that IS what discussion is for. To take ideas from inside our heads or inside conceptions and see how they sound in the real world , then refine, or remove them, roughly speaking.

                                                                                   

                                                                                  Scott Chichelli quoted an external document which conluded with this:

                                                                                  The simple focus point here is exceed the bit rate of the lowest compression codec used by several times. Then expect multiple disks for every set of 4 to 5 tracks or so to handle the requests. The Render/export times are also effected by the disks because now you have multiple read and write requests to assign. This is why it's better to split the read and write load between 2 disks or volumes depending on whether you require smaller or larger raid arrays.

                                                                                   

                                                                                  This is all very clear for me until the last sentence. I originally assumed it was recommending separating the 'reading' from the 'writing' but reading it a few more times I believe it is in fact saying that due to the extra drain on bandwidth it may be "better to split the read and write load between 2 disks or volumes depending on whether you require smaller or larger raid arrays."

                                                                                   

                                                                                  My initial assumption was that adding a new volume for export was to prevent corruption of the written data due to reading then writting then reading then writting etc, but if this is not the case (?) then, assuming there is no lack of bandwidth, there is no requirement to do that. Further, it is in fact, squandering or wasting bandwith which might be utilized for the 'reading' of timeline media!

                                                                                   

                                                                                  While the export volumes disk/s are incorporated in the main array/volume the bandwidth can be distributed to reading requirements, if the disk/s are outside of the array, it can offer nothing productive towards the editing process until render/export time.

                                                                                   

                                                                                  Take it to the Nth degree again: what if the project was so demanding (Extreme requirements such as 4K 16-bit DPX files) that editing was requireing 100% of bandwidth? When it came time to render/export, what would the result be? Would there be a downside??

                                                                                  • 38. Re: more than two volumes - real-world benefits?
                                                                                    Jeff Bellune Level 6

                                                                                    My system originally came configured with 2x 2-disk RAID0s, a 3-disk RAID5, a system/OS drive and an "all other data" drive.  I had one RAID0 set up for source media and one set up for projects and exports.  As time went by and I replaced the drives, I re-configured the 2xRAID0s to a single, 4-disk array that now holds media, projects and exports/renders.

                                                                                     

                                                                                    Why did I do that?  Typically, I either have highly-compressed source video, or I export highly-compressed video.  That means that at one end of the editing pipeline or the other, almost any disk data throughput will be fast enough during the time the CPU chews on the footage.

                                                                                     

                                                                                    I have HD footage, both my own and from clients and 3rd parties, that is uncompressed or losslessly compressed.  That HD footage, which needs very little CPU time to decompress, really benefits from a 4-disk RAID0.  So if that  footage is placed on my timeline, I have a very smooth editing experience because of the high I/O throughput of my RAID0.  Additionally, I rarely export that HD footage to uncompressed or lossless, so a RAID0 is overkill for the exporting task.

                                                                                     

                                                                                    I also have HD source footage that is highly compressed (AVCHD), that I will export to lossless or uncompressed for archival or collaborative purposes, and in that case it is the export that benefits most from the RAID.  But  even in that case, with some formats and codecs the CPU is so involved in the export that a single 7200 rpm SATA disk would probably provide the same performance.

                                                                                     

                                                                                    I'm happier with my 4-disk, single-volume RAID0 than I was with the 2x 2-disk RAID0.  NB: that may be due as much to the fact that the 2xRAIDs were only 2-disk arrays; if each had been 4-disk arrays I may have preferred the setup with the extra RAID.

                                                                                     

                                                                                    I have the 3rd volume, the "all other data" drive, for ease in backing up and syncing files between my desktop and my laptop -- I travel a lot and the 3rd volume makes my life easier.  In that respect, you could say it's "more efficient" than not having it.  Finally, my data drive and my RAID0 are backed up incrementally and continuously to the RAID5.

                                                                                     

                                                                                    So I have 4 volumes that I use all the time, down one from the 5 volumes I started with.  I wouldn't say my editing and travel habits are "extreme", but I probably do travel more than most people.  So if you ditch the extra data volume that is used efficiently for syncing with a laptop (because you don't need to sync files very often), and if you ditch the RAID5 backup volume (because the hypothesis is that backup is already taken care of over the network), you could reduce my system down to 2 volumes and have a very happy and productive editing experience.  That said, you would still need more than 2 physical disks, simply because HD is everywhere now, and at least a 2-disk RAID0 is necessary if you ever start pushing uncompressed or lossless HD frames around.

                                                                                     

                                                                                    Jeff

                                                                                    1 person found this helpful
                                                                                    • 39. Re: more than two volumes - real-world benefits?
                                                                                      Scott Chichelli Level 3

                                                                                      Jeff Bellune wrote:if each had been 4-disk arrays I may have preferred the setup with the extra RAID.

                                                                                      i think that would be the case yes..

                                                                                      but also requres a raid card.

                                                                                       

                                                                                      eventually we will have benchmarks until then all this discussion is fairly moot without proof of statement be it for or against.

                                                                                       

                                                                                      Scott

                                                                                      ADK

                                                                                      1 2 Previous Next