16 Replies Latest reply on Jun 30, 2012 9:29 PM by RFDPiper

    Help with HDD lingo for a dummy

    RFDPiper Level 1

      Hi folks, obviously a new guy here.

      I'm hoping someone can clear up the lingo when talking about HDD setup.  Specifically, what is meant by

       

      "scratch"

      "pagefile"

      "media cache"

      "previews"

       

      I'm learning computers as I go, and I built my computer for the Production Premium suite.  My set up currently is I have 2 300GB Velociraptors in RAID 0 as my C drive where I keep everything (OS, programs, media, renders, etc.) and a single 1TB HDD for backup.  Before any of you have an aneurysm, I know that this is the worst possible set up.  That's why I'm trying to change it.

       

      I have a good idea for a new setup plan by reading these forums, but I'm having a little bit of difficulty following along with the lingo.

       

      My plan is to go with:

      Two 300GB Velociraptors in RAID 0 for my raw media from my video camera.

      A single 1TB HDD for my OS and programs (I know that's excessively large, but I already have one, so it's free).

      Two 300GB Velociraptors in RAID 0 for my saved files (i.e. where I would save my AE or PR projects, and my final renders to).

      2TB RAID 1 networked storage/backup

       

      I'm sure the above are things I already know, but I don't know the words.  Can anyone translate these into dummy language?  (For example: Media = the recordings you got from your camera)

       

      Thanks,

      Bryan

        • 1. Re: Help with HDD lingo for a dummy
          Jim_Simon Level 9

          Previews refer to files generated while editing to make realtime viewing possible.

           

          Media Cache are files used by Premiere Pro internally.  They're created automatically upon import, and if they ever get deleted, PP will simply recreate them when it needs to.

           

          Both fall under the heading of Scratch, which comes from the term "scratch pad", which is scrap paper to work and take notes with, but not the final draft.

           

          Page file (also called swap file) is hard drive space used by Windows when the available memory has become full.  Windows will write unneeded data in memory to a hard drive so it has room to load the needed data.  With 12GB or more of memory, you probably won't be using the swap very much.  Still, it's a good idea to have it, because Windows can crash if it ever needs it and it's not there.  I know Harm recommend setting the page file manually to a disk other than the C: drive.  I disagree with the idea.  Because the page file isn't used very often in a decent edit system with lots of memory, you're perfectly fine leaving the page file where it is, managed by Windows.  Most people won't see any benefit by tampering with it, and may even mess things up if not done right.

           

          RAID 0 is a bad idea for any edit system.  If one drive fails, you lose all data on both drives.  Here's a better plan:

           

          C: System (Windows and Program installation) - 300GB

          D: Project Files (Premiere Pro, After Effects, Encore, etc. as well as any still images and audio files used in the project) - 300GB

          E: Media (from the camera) - 1TB

          F: Scratch (previews, media cache) - 300GB

          G: Export (what comes out of Premiere Pro) - 300GB

          I: Archiving (Projects and media, as well as final export) - 2TB network drive

          1 person found this helpful
          • 2. Re: Help with HDD lingo for a dummy
            Harm Millaard Level 7

            I second Jim's suggestion, with one exception, the pagefile.

             

            Windows requires a pagefile and if one is not defined, it will create one on the C: drive. First of all, Windows is dumb enough to make a dynamically managed file that is up to 1.5 times the size of installed memory. On a 12 GB system that means a 18 GB max. pagefile and on a 24 GB system even a max. 36 GB pagefile. Because of its dynamic nature it will cause fragmentation and thus performance degradation. On conventional disks the fastest part of the disk is the first file created on that disk, which makes it very sensible to put it on a physically different disk than the OS disk (it spreads the load for the OS) and to make it the first file on that different disk. If you also make it static, instead of dynamic, the size never changes and will thus not become fragmented. A nice size would be around 12 GB on 12 - 24 GB systems.

             

            For SSD boot disks placing the pagefile on the SSD is a bad idea because of the write degradation that all SSD's suffer from and this is not taking into account the high price per GB of SSD's where a dynamic pagefile is a costly 'asset' to have on the SSD.

             

            For simplicity I would put a static pagefile of 12 GB on the D: drive as the very first file, following Jim's disk allocation.

             

            @ Jim: Setting up a static pagefile on another drive is not rocket science. In fact I think even my wife could manage that.

            1 person found this helpful
            • 3. Re: Help with HDD lingo for a dummy
              RFDPiper Level 1

               

              RAID 0 is a bad idea for any edit system.  If one drive fails, you lose all data on both drives.  Here's a better plan:

               

              C: System (Windows and Program installation) - 300GB

              D: Project Files (Premiere Pro, After Effects, Encore, etc. as well as any still images and audio files used in the project) - 300GB

              E: Media (from the camera) - 1TB

              F: Scratch (previews, media cache) - 300GB

              G: Export (what comes out of Premiere Pro) - 300GB

              I: Archiving (Projects and media, as well as final export) - 2TB network drive

              I'm a bit confused.  Why is everyone so down on RAID 0?  Isn't it much much faster?  As far as losing information, if you lose the E drive in your example above, you lose all your media too.  As far as I understand it RAID 0 has no disadvantage over a single disk other that the cost of the setup, but has a tremendous advantage as far as speed (I'm new obviously, so please correct me if I'm wrong).

               

              Jim, wouldn't the setup you suggested be slower than the one I suggested?  That's not a criticism, I'm just trying to learn.

               

              P.S.  for further consideration:  I purchased my two new drives, and I got two 450GB 6mbs Velociraptors instead of the 300GB 3mbs like my old ones since they were the same price, and yes, I do have two 6mbs ports available on my motherboard.

               

              So now I have:

              2) 300GB 3mbs Velociraptors

              2) 450GB 6mbs Velociraptors

              1) 1TB WD Caviar Black

              and a 2TB RAID 1 NAS

              • 4. Re: Help with HDD lingo for a dummy
                Jim_Simon Level 9
                @ Jim: Setting up a static pagefile on another drive is not rocket science.

                 

                No, but neither is it necessary, in my view.  Editing performance isn't likely to change at all by moving it.  And Windows 7 takes care of fragmentation on it's own.  (Or you can use a third party tool for this.)

                 

                The only scenario I'd agree that moving it is a good idea is when the boot drive is an SSD (which neither of us generally recommends anyway.)

                • 5. Re: Help with HDD lingo for a dummy
                  Jim_Simon Level 9

                  Why is everyone so down on RAID 0?  As far as losing information, if you lose the E drive in your example above, you lose all your media too.

                   

                  True.  But with one drive, you only worry about that one drive failing.  With a RAID 0, you have to worry about either drive failing.  So the chances of losing the data increase with every drive added to a RAID 0.

                   

                  As far as speed goes, you won't need it when using most standardized camera formats.  A single modern drive is plenty fast enough for even multiple streams.

                   

                  Where the speed is really needed is with Uncompressed or 4K formats (like form a RED camera).  But when that speed is needed, I recommend a RAID 3 using an Areca controller card.  It offers most of the speed of a RAID 0, but with built in data security in the event of a drive failure.

                  • 6. Re: Help with HDD lingo for a dummy
                    RFDPiper Level 1

                    You guys are very helpful, thank you.

                     

                    Sorry for the redundancy, but before I set my system up, I would like to confirm something with you because I do in fact work with uncompressed footage if that does indeed make a difference.  I use an HDV camera, but I do a lot of greenscreen work, so I capture the raw uncompressed video from my camera to a BlackMagic Hyperdeck Shuttle and it provides fantastic keys.  I don't know how big, but I know that 20 minutes of video completely fills the Hyperdeck's 256GB hard drive.  So this is where I think that RAID 0 would be very helpful, but I'll defer to you guys for the best advice.

                     

                    With my five HDD bays, here are the options I see:

                     

                    My suggestion:

                    C: One 1TB WD Caviar Black  -   OS and Programs

                    D: Two 300GB 3mbs Velociraptors RAID 0  -  Media, projects

                    E: Two 450GB 6mbs Velociraptors RAID 0  -  Previews, media cache, exports

                    Plus a 2TB RAID 1 NAS   -   Backup/archive

                     

                    Jim's suggestion:

                    C: System  - 300GB

                    D: Project Files - 300GB

                    E: Media - 1TB

                    F: Scratch - 300GB

                    G: Export - 300GB

                    I: Archiving - 2TB network drive

                     

                    Harm's suggestion (from the spreadsheet, 5 disk setup)

                    C: OS, programs, pagefile (Harm said with lots of memory I can use the array for the pagefile, is 24GB a lot?)

                    D: (RAID 5) - Media, projects, media cache, previews, exports  (then that begs the question, can I RAID 5 two 300GB 3gbs and two 450GB 6gps HDDs, or do they all have to be the same?)

                    (Plus my 2TB NAS for archive)

                    • 7. Re: Help with HDD lingo for a dummy
                      Harm Millaard Level 7

                      Ingesting over HD-SDI or even over the poor man's version, HDMI, requires huge amounts of bandwidth, up to 1.485 Gbps and that is only doable with raid arrays. However JBOD is not advisable and using different disks in a raid0 mens that the size of the smallest disk is all that is used for the array and the rest is not used. In your example mixing 300 and 450 GB disks in an aid0 means that your storage capacity is limited to 4 x 300 GB and is far from optimal because of differences between the two kinds of disks.

                       

                      For all arrays, stick to the same brand and model for all the member disks.

                       

                      With HDV you have the compressed tapes with the original footage, albeit in 4:2:0 colorspace, as a backup and thus you can use 2 x 450 aid0 for your media and projects and the other 2 x 300 aid0 for media cache, pagefile and exports. Even then, this may not be enough for uncompressed. I think that with your workflow a dedicated raid controller and a large number of disks in a parity raid is the better solution. Say a setup like:

                       

                      C: OS & programs

                      D: 2 x 300 aid0 for pagefile, media cache, previews (sustained transfer rate around 220 MB/s)

                      E: 5 or 6 x 1000 Raid3/5 for media, projects and export (sustained transfer rate around 330 with 5 to 410 MB/s with 6 drives)

                      • 8. Re: Help with HDD lingo for a dummy
                        RFDPiper Level 1

                        E: 5 or 6 x 1000 Raid3/5 for media, projects and export

                        I thought it was a sin to read/write from the same disk (i.e. media/export)?  Is that not the case?

                        • 9. Re: Help with HDD lingo for a dummy
                          Harm Millaard Level 7

                          There are some exceptions, but most people, when having finished their project and start exporting think "I'm glad it is finished now. I'll tell my wife/Managing Director/client that it is finished and rendering out to Encore or whatever. I'll grab a beer while I wait for this process to finish to celebrate this milestone."

                           

                          Remember exporting is the single last step in a project (apart from the authoring in Encore) and happens only once in a project. You may gain a few timeslices by exporting to a different disk if you have one, but if this means you have to reduce your raid to one less disk, the penalty in performance is bigger during the editing phase than the gain during export.

                          • 10. Re: Help with HDD lingo for a dummy
                            flyingfish4 Level 1

                            Thanks again Harm and everyone for some good education. 

                            I've read different things about exporting and I'm a bit confused. 

                            Is the CPU the bottleneck in exporting, not the speed of the drive? 

                            I think I read that you could even use an external USB 2.0 drive to export to? 

                            If exporting to h.264, does a RAID 0 export drive not increase the speed of the export with an i7 2600K (with either mild or no overclocking) & 16GB RAM? 

                            Can the CPU not go as fast as a RAID 0 can allow?

                            Thanks again for everyone's help.

                            • 11. Re: Help with HDD lingo for a dummy
                              RjL190365 Level 4

                              RFDPiper,

                               

                              In your case, you will need multiple 1TB enterprise-class hard drives and a discrete hardware RAID card. None of the hard drives currently in your posession are suitable for parity RAIDs such as the one Harm and I recommend.

                               

                              My recommended setup would be as follows:

                               

                              C: Two 450GB 6 Gbps Velociraptors in RAID 1 (450MB effective total capacity) for the OS and programs

                              D: Two 300GB 3 Gbps Velociraptors in aid0 for pagefile, media cache and previews

                              E: Multiple 1TB enterprise-class drives, such as the Western Digital RE4 or Seagate Constellation series hard drives connected to a discrete hardware RAID controller card set to RAID 5 (or RAID 3 if using an Areca card)

                               

                              Plus, your existing 1TB WD Black and 2TB NAS drives for backups.

                              • 12. Re: Help with HDD lingo for a dummy
                                RjL190365 Level 4

                                flyingfish4,

                                 

                                (R)aid0 will not speed up the H.264 encoding at all. This is because the maximum bitrate for H.264 video content is far below even the minimum sequential transfer speed of a single hard drive. Plus, H.264 encoding performance depends almost entirely on the CPU.

                                • 13. Re: Help with HDD lingo for a dummy
                                  flyingfish4 Level 1

                                  Thanks RjL.

                                  • 14. Re: Help with HDD lingo for a dummy
                                    RFDPiper Level 1

                                    This is not the case for me however, as I use both AE and Premiere Pro, so rendering from AE to Pr is a constant part of my workflow.  In this case I probably should have separate drives for media and renders right?

                                    • 15. Re: Help with HDD lingo for a dummy
                                      Jim_Simon Level 9

                                      That would speed things up.

                                       

                                      But why render out?  Why not use Dynamic Link?

                                      • 16. Re: Help with HDD lingo for a dummy
                                        RFDPiper Level 1

                                        Because I need my Pr timeline to play back in real time so I just render out my AE projects as lossless files and drop them in Pr.  Dynamic link doesn't allow realtime playback