13 Replies Latest reply on Oct 8, 2009 2:25 PM by Harm Millaard

    Scratch Disk Questions

    ambisinistrous

      Only pagefile, scratch/renders, and program files on this drive? Any reason not to do some file storage on this drive, too? Or what about other programs that never run simultaneously with editing apps?

       

      Is it acceptable to have my main media storage drives in a RAID array, but use a single unraided SATA drive for scratch?  Or should one move the scratch disk from a single drive to RAID in these circumstances?

        • 1. Re: Scratch Disk Questions
          Harm Millaard Level 7

          What disk setup are you contemplating? Number of disks, interface, size, speed?

          • 2. Re: Scratch Disk Questions
            ambisinistrous Level 1

            My questions are more about general principles, with the only specific that you have 1 striped RAID array and 1 non-raided scratch drive.  But I can throw out more specifics for the sake of argument.

             

            Let's say a single 500 GB Seagate Barracuda 7200.12 (7200 rpm, SATA) for scratch/pagefile/renders. Then 2x 1 terrabyte drives from the same line in a RAID 0 array using the motherboard's RAID controller.

             

            This makes me think of another question. What's an appropriate ratio of scratch/render storage to media storage? For instance, if you had 200 GB of HD source footage, how much scratch/render would you allocate for that?

            • 3. Re: Scratch Disk Questions
              Jim_Simon Level 9

              Generally speaking, here is a good minimum/starting point.

               

              C: OS and Programs

              D: Projects/Scratch

              E: Media

               

              Don't worry about the swap.

               

              Now from there, you can do a lot of tweaks.  But the above is a good basic design for an NLE disk system.

              • 4. Re: Scratch Disk Questions
                ambisinistrous Level 1

                Thanks for the responses guys. I'm afraid I maybe muddled my questions. Let me try again

                 

                1. Is it worthwhile to have a non-RAIDed, single-drive for scratch/renders, while using RAID striped drives for media? Or should you just keep scratch/renders on the single RAID array along with the main media files?  To put it another way, if you only have 1 striped RAID array, which you use for media, does it make sense to use a separate single drive for scratch/renders?
                2. What's a good ratio for media drive capcity to scratch drive capacity? For instance, if you have 2 TB of media storage, how big do you want your scratch/render drive to be?
                • 5. Re: Scratch Disk Questions
                  Harm Millaard Level 7

                  1. To get the best speed from your disk setup, you need to make sure that disk activity is spread across as many disks as you have. When editing, your OS & programs need to do housekeeping, need to load plug-ins, need to load render engines, etc. This all happens on your C drive. At the same time it needs to read your media (from the E drive) and it needs to write and read preview files and keep the media cache up to date (from your D drive). At the same time your pagefile need to be accessed. I realize this is not all the activity that takes place on all your disks, but you get the picture that a lot of different reads and writes need to occur. The more you spread the load over all your disks, the better. When you have two disks and consider striping them, realize that you can only use Aid0 to get some performance gain. However, a valid question is whether that performance gain is worth the risk of striping and if it is not better to leave them as separate disks. Of course this argument does not hold with many more disks in an array where you can maintain redundancy, then raids are a definite advantage.

                   

                  2. I'm not aware of a rule of thumb for this ratio, but with storage so cheap I would use at least 500 G. Keep in mind that all mechanical SATA disks show decreasing performance when the fill rate goes up. So with current prices, you can make life easier by keeping all your disks pretty large with lots of space available. That is the reason I have 16 x 1 TB disks in my system.

                  • 6. Re: Scratch Disk Questions
                    Ferdie2 Level 1

                    Couldn't you also clear out your D drive when projects are finished?  Further diminishing the need for a large capacity Scratch drive.

                    • 7. Re: Scratch Disk Questions
                      Harm Millaard Level 7

                      Why are you so worried about disk space? Get several 1 TB disks and edit along. At current prices for 1 TB disks around $ 80 apiece, what is stopping you?

                      • 8. Re: Scratch Disk Questions
                        Ferdie2 Level 1

                        I'm not concerned with disk space on the Scratch drive.  But wouldn't the rendering process be faster with a 10k drive as opposed to a 7.2k drive?

                        • 9. Re: Scratch Disk Questions
                          Harm Millaard Level 7

                          Keep in mind that the Velociraptor has only 16 MB cache, whereas a 1 TB disk has 32 MB cache. Also keep in mind that on a 150 GB disk 100 GB of data amounts to a fill rate of around 70%, while on a 1 TB disk that would only amount to 10% fill rate. Comparing both disks when empty, yes, the Velociraptor is faster, but when filled up to 70% is may well be slower than a 1 TB disk filled to 10%.

                           

                          Look at these charts:

                           

                          HD Tach C.jpg

                           

                          The spikes are caused by the cache being depleted. With 100 GB data on the disk, the sequential read speed is around 100 MB/s

                           

                          Now look at the graph from a 1 TB disk:

                           

                          HD Tach B.jpg

                           

                          With the same 100 GB data on this disk the sequential read speed is around 115 MB/s

                           

                          Now look at the graph for a 2 disk raid0 array with 2 x 1 TB disks:

                           

                          HD Tach D.jpg

                           

                          With up to 900 GB of data the sequential read speed stays at around 200 GB/s.

                           

                          And the last one, a 12 x 1 TB disk raid30 array shows these figures:

                           

                          Areca_HDTach1.jpg

                           

                          Across the board this delivers a sequential read speed of over 800 MB/s (average 853 MB/s) and no deterioration with increasing fill rates.

                           

                          Although these graphs only indicate sequential read speeds, the same pattern applies to write speeds.

                          • 10. Re: Scratch Disk Questions
                            Ferdie2 Level 1

                            Excellent data.  So, to summarize:

                             

                            RAID30 > RAID0 > 1TB 32MB Cache > 150GB 16MB Cache

                             

                            (More q's on my own thread.)

                            • 11. Re: Scratch Disk Questions
                              Harm Millaard Level 7

                              Unfortunately, Ferdie, it is not so simple.

                               

                              Raid performance depends heavily on the number of disks in the array. That is the overriding factor. Next is the raid level. That depends heavily on the raid controller used and the degree to which REDUNDANCY is required. Next is the cache memory used by the raid controller.

                               

                              We all know that disks sometimes fail. We hope they do not, but in practice they do from time to time. When that happens, you lose all data on that disk, period. When you stripe disks the chances of that happening are multiplied by the number of disks in the array. When you have 2 disks in the array, chances of failure are doubled, with 3 disks, the chances are tripled, and so on. That is why I call a Raid0 an AID0 since there is no redundancy.

                               

                              When you use a real Raid (with redundancy) with level 3, 5 or 6, you can have one or more disks fail and you will not lose data. The disk that went hayward will be rebuild upon replacement and it will be done in the background, so you can continue working. You can even use hot-spares. In case a disk fails, it's role in the array will be automatically taken over by the hot-spare disk.

                               

                              To learn more about RAID, look here: http://www.raid.com/04_00.html

                               

                              A RAID30 is a striped array of 2 or more Raid3's.

                              • 12. Re: Scratch Disk Questions
                                Ferdie2 Level 1

                                Harm Millaard wrote:

                                 

                                Unfortunately, Ferdie, it is not so simple.

                                 

                                Yes, I was generalizing based on the scenario given.  I know there are numerous variations of every aspect.

                                 

                                When you use a real Raid (with redundancy) with level 3, 5 or 6, you can have one or more disks fail and you will not lose data. The disk that went hayward will be rebuild upon replacement and it will be done in the background, so you can continue working. You can even use hot-spares. In case a disk fails, it's role in the array will be automatically taken over by the hot-spare disk.

                                 

                                To learn more about RAID, look here: http://www.raid.com/04_00.html

                                 

                                A couple of nits.  RAID1 is redundant, isn't it?  I'm not familiar with levels 2 and 3, but RAID 4 has a unique benefit.  Since it has a dedicated parity drive, you can add disks to the set on the fly.  I used to be a Netapp admin, and this was one of the benefits that sold us on RAID4.  Nowadays, they have diagonal parity and I just don't understand anymore.

                                 

                                Also, you can't lose more than one disc at a time, at least not while in degraded mode.  If you lose a second drive while a hot spare is being created (sometimes 8 hours or more), your volume is toast.

                                • 13. Re: Scratch Disk Questions
                                  Harm Millaard Level 7

                                  Correct. If you use raid levels 3, 4 or 5 only one disk can fail. If you lose more disks at the same time, bye, bye. With level 6 you can have 2 disks fail at the same time without negative impact. You can recover from a 2 disk failure in raid6. BTW, Raid4 is hardly used anymore because of it's lousy write performance.

                                   

                                  In practice, what are the chances of 2 disks failing at the same time? I have had a problem on my server with a 6 disk raid5 array where 2 disks were showing intermittent problems. I first replaced one disk, rebuilt, then replaced the second disk, rebuilt again and had no data loss at all. On my NAS I use a 7 disk raid6 so I can have 2 disks fail at the same time (this was rather necessary since 5 out of 7 Seagates failed in the first 3 months). On my editing machine I still contemplate adding a hot-spare to each of the raid3 arrays, so that I have double redundancy, but it is slightly troublesome to fit 19 disks in my case next to two BR burners and it precludes the use of 4 SSD's in a single 5.25" slot.