7 Replies Latest reply on May 21, 2010 9:18 AM by Harm Millaard

    Best Config of Drives

    FelixUnderwood Level 2

      I feel like I'm in first grade all over again, yet I build my own editing rigs. Clearly a lack of understanding, which leads to my ignorance in storage I/O. I've heard it suggested using RAID 0 for a fixed pagefile of either 8 or 12 GB, media cache, previews and rendered files. And this same person suggested a single drive for media. Four questions: 1: when I'm playing back, say, a quad split, real time with MPE, aren't I taxing the media drive for the most throughput? Is the RAID 0 being taxed at all, being that I'm not using rendered files? 2. When playing back a rendered timeline, isn't the data stream basically one layer, since all of the layers are being rendered to one, so I would think this could be the single drive...  3. What's the difference between Previews and Rendered Files? I always thought they were synonomous.  4. What is media cache and what's the data I/O requirements like? Should it get it's own drive or can it share with another data stream? Please note that when I say "I would think" that's it's ignorance that causes me to think this way and that's why I'm asking someone who clearly understands the data flow. I apologize in advance if any of you are repeating yourself for the 13,000th  time from other posts I haven't read, but I bet a plethora of other Forum participants will benefit from the answers to these 4  questions answered together. Sending the various media streams to the wrong drives or together with other data streams that shouldn't be shared to the same drive can be a major bottleneck. Perhaps you can list your drive setup and what you have delegated to each drive. Since I'm the one asking for help, I promise not to chastise you for the answer you give! (Actually I wouldn't chastise you even if you were the one asking for help...)

        • 1. Re: Best Config of Drives
          Harm Millaard Level 7

          Felix,

           

          Start with the basic fact that sequential reading is the fastest method of getting data off a disk. Random writing is the slowest method of getting data on a disk, with sequential writing next.

           

          When one renders a timeline, with many small files, this difinitely is writing, but in some cases may be closer to randiom writing, than sequential, depending on the length of transitions and effects.

           

          When scrubbing through a timeline, this is reading and again depending on the length of yellow or green areas in the timeline, is sequential reading for yellow and can approach random reading for the rendered areas.

           

          Each situation must be considered in relation to the number of disks avaiable and the intended raid configuration. With a two disk raid0, the risk of data loss is twice as high as a single disk. What are data that do not really matter is they are lost, well, that are data that are easily recreated, like the pagefile and rendered files, so it makes perfect sense, due to the nature of accessing these data in an editing flow and from a security aspect, to use these on a raid0.

           

          Media files, especially with tapeless workflows, would often be disastrous to lose. In a tapebased workflow one can recapture, but not in a tapeless workflow, so safety of data is much more important and should for that reason alone not be put at risk on a raid0.

           

          If one has a limited number of disks, let's say only 3, apart from the OS disk, it often is even better not to raid at all, but use them as single disks. The situation changes when one has, let's say 5 disks. Then a 5 disk raid5 may be the more attractive solution.

           

          You have to see my answer, that you alluded to, in the light of your question, asking which of two options was the better one.

           

          Preview/render was used because some people only talk about render files, others about preview files, but they are one and the same.

           

          I realize that my disk setup is rather uncommon, but this is my setup:

           

          C: OS and programs, Velociraptor

           

          D: Pagefile, temp files, media cache, miscellaneous, 2 x 1 TB raid0

           

          E: Projects, media, render files, 12 x 1 TB raid30

           

          A: Stock footage and audio, 1 TB

           

          B: Exports, 1 TB

           

          Hope this explains it a bit, and no problem for asking.

          • 2. Re: Best Config of Drives
            FelixUnderwood Level 2

            Wow. I just gained a lot of knowledge in one answer, which was precisely what I was looking for and hope others will find useful as well. Many thanks! That obviously took some time and I thank you very much for sharing your knowledge, as you so generously often do. Another batch of questions: 1. Okay, let's say I back up my media religiously, taking some of the added risk of RAID 0 out of the picture... Would it then make sense to make the RAID 0 the media drives and a single drive the one for the Page/Media Cache/Rendered files? It's that throughput issue I'm concerned about with using only one drive for media. As you stated before, MPE has limited utility if the bottleneck is at the HDs. 2. Over the years, have you had better luck with Seagates or WDs (or perhaps even another brand)? 3. WD has a 10K 600Gb drive. Your thoughts on RAIDing 3 of these in 0.  I'm about to build a new editing rig and the media config is the only aspect I haven't signed off on yet. I guess, Harm, I'm trying to figure out the best way to get the best throughput with say, 4 HDs. Oh yeah, this might help: I'm currently editing primarily with HDV but plan on migrating over to a Panasonic P2 camera that will record in AVC-I and DVCPro HD (and that's a totally different discussion!)

            • 3. Re: Best Config of Drives
              Harm Millaard Level 7

              Felix,

               

              To start off with various brands of disks, I haven't used WD's for ages, but the Caviar Black has a good reputation for single disks, whereas the RE4 has a good reputation for raided disks. With Seagates I have had lousy experiences. I had 7 Seagate 7200.11 1.5 TB disks in a NAS and within a couple of months 5 out of 7 died. Now the 7200.11 series was notoriously bad and the 7200.12 series are alledgedly much better. In my system I have 16 Spinpoint F1's, and in the past year I had to exchange two of them because of intermittent problems. The F3 series has a good reputation.

               

              If you were to opt for a 3 disk raid0 and put everything, from projects and media to media cache and rendered files on it, to make it easy on yourself in an organizational sense, you would have fantastic thruput. But that presupposes, as you said, meticulous backups of your media files.

               

              One solution might be to use three Velociraptors in raid0 (1.8 TB) and use a backup 2 TB disk, where you can store things like downloads, stock footage, and other miscellaneous files as well.  Just make sure you set your Windows environment variables for TEMP and TMP to the raid.

               

              If the 2 TB disk is an internal SATA or even an eSATA external one, you can locate the pagefile there as well, especially if you have 12 G memory.

               

              Your material is pretty easy on the CPU, so I understand your worries about disk thruput. Maybe these remarks help.

              1 person found this helpful
              • 4. Re: Best Config of Drives
                FelixUnderwood Level 2

                Harm, to hear you say that I'd have fantastic throughput is good enough for me! Unless you think it's overkill for what I'm doing (mainly 30 sec. spots soon to be shot in AVC-I or DVCPro HD (same basic data requirements) with 4-12 layers, I think I'll go with the 3 x 600Gb 10K drives from WD. The last remaining component that I haven't decided on yet because of your advice in another thread is to wait and see what NVidia comes up with next month in regards to their new Quadro cards. Hopefully it will be worth the wait...oh, and then there's that validation lag! I may go another video card route temporarily. Many thanks again for taking so much time to steer me, and hopefully others, in the right direction for CS5.

                • 5. Re: Best Config of Drives
                  Harm Millaard Level 7

                  Felix,

                   

                  I still owed you one answer, what are media cache and media cache files?

                   

                  Media cache files are PEK, CFA and index files, in short all the files  created on import by indexing, conforming and creating peak files for  the wave form display.

                  The media cache contains  pointers to these files, AFAIK.

                  Access to these files and the speed with which a project loads is very much dependent on the speed of the disk. It also impacts the speed of display when zooming the timeline in or out or when scrolling horizontally throught the timeline.

                   

                  Somebody please correct me if I'm wrong.

                  • 6. Re: Best Config of Drives
                    FelixUnderwood Level 2

                    Harm, you did answer all of my questions. I can't thank you enough. Since I'm going from a dual quad core to a single six-core, I'll trade out the extra CPU with it's heat and power requirements for the 3 Velociraptor 600s in a RAID 0 and I will have two 2Tb drives left over. Better get a RAID card! I'll check out your posts for recommendations on that, unless you haven't made a recommendation on a Sata III card yet. Thanks again. You've put a lot of time into this post alone!

                    • 7. Re: Best Config of Drives
                      Harm Millaard Level 7

                      SATA III controller cards are like: "Serendipity is like looking for a needle in a haystack and finding the farmer's daughter".

                       

                      You will hard pressed to find one currently, but Areca anncounced the Areca ARC-1880iX-12/16/24 range a couple of months ago and they may start to begin appearing on the market.