Skip navigation
Currently Being Moderated

Help with HDD lingo for a dummy

Sep 27, 2011 12:02 PM

Hi folks, obviously a new guy here.

I'm hoping someone can clear up the lingo when talking about HDD setup.  Specifically, what is meant by

 

"scratch"

"pagefile"

"media cache"

"previews"

 

I'm learning computers as I go, and I built my computer for the Production Premium suite.  My set up currently is I have 2 300GB Velociraptors in RAID 0 as my C drive where I keep everything (OS, programs, media, renders, etc.) and a single 1TB HDD for backup.  Before any of you have an aneurysm, I know that this is the worst possible set up.  That's why I'm trying to change it.

 

I have a good idea for a new setup plan by reading these forums, but I'm having a little bit of difficulty following along with the lingo.

 

My plan is to go with:

Two 300GB Velociraptors in RAID 0 for my raw media from my video camera.

A single 1TB HDD for my OS and programs (I know that's excessively large, but I already have one, so it's free).

Two 300GB Velociraptors in RAID 0 for my saved files (i.e. where I would save my AE or PR projects, and my final renders to).

2TB RAID 1 networked storage/backup

 

I'm sure the above are things I already know, but I don't know the words.  Can anyone translate these into dummy language?  (For example: Media = the recordings you got from your camera)

 

Thanks,

Bryan

 
Replies
  • Currently Being Moderated
    Sep 27, 2011 2:14 PM   in reply to RFDPiper

    Previews refer to files generated while editing to make realtime viewing possible.

     

    Media Cache are files used by Premiere Pro internally.  They're created automatically upon import, and if they ever get deleted, PP will simply recreate them when it needs to.

     

    Both fall under the heading of Scratch, which comes from the term "scratch pad", which is scrap paper to work and take notes with, but not the final draft.

     

    Page file (also called swap file) is hard drive space used by Windows when the available memory has become full.  Windows will write unneeded data in memory to a hard drive so it has room to load the needed data.  With 12GB or more of memory, you probably won't be using the swap very much.  Still, it's a good idea to have it, because Windows can crash if it ever needs it and it's not there.  I know Harm recommend setting the page file manually to a disk other than the C: drive.  I disagree with the idea.  Because the page file isn't used very often in a decent edit system with lots of memory, you're perfectly fine leaving the page file where it is, managed by Windows.  Most people won't see any benefit by tampering with it, and may even mess things up if not done right.

     

    RAID 0 is a bad idea for any edit system.  If one drive fails, you lose all data on both drives.  Here's a better plan:

     

    C: System (Windows and Program installation) - 300GB

    D: Project Files (Premiere Pro, After Effects, Encore, etc. as well as any still images and audio files used in the project) - 300GB

    E: Media (from the camera) - 1TB

    F: Scratch (previews, media cache) - 300GB

    G: Export (what comes out of Premiere Pro) - 300GB

    I: Archiving (Projects and media, as well as final export) - 2TB network drive

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 27, 2011 3:29 PM   in reply to Jim Simon

    I second Jim's suggestion, with one exception, the pagefile.

     

    Windows requires a pagefile and if one is not defined, it will create one on the C: drive. First of all, Windows is dumb enough to make a dynamically managed file that is up to 1.5 times the size of installed memory. On a 12 GB system that means a 18 GB max. pagefile and on a 24 GB system even a max. 36 GB pagefile. Because of its dynamic nature it will cause fragmentation and thus performance degradation. On conventional disks the fastest part of the disk is the first file created on that disk, which makes it very sensible to put it on a physically different disk than the OS disk (it spreads the load for the OS) and to make it the first file on that different disk. If you also make it static, instead of dynamic, the size never changes and will thus not become fragmented. A nice size would be around 12 GB on 12 - 24 GB systems.

     

    For SSD boot disks placing the pagefile on the SSD is a bad idea because of the write degradation that all SSD's suffer from and this is not taking into account the high price per GB of SSD's where a dynamic pagefile is a costly 'asset' to have on the SSD.

     

    For simplicity I would put a static pagefile of 12 GB on the D: drive as the very first file, following Jim's disk allocation.

     

    @ Jim: Setting up a static pagefile on another drive is not rocket science. In fact I think even my wife could manage that.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 19, 2011 2:19 PM   in reply to Harm Millaard
    @ Jim: Setting up a static pagefile on another drive is not rocket science.

     

    No, but neither is it necessary, in my view.  Editing performance isn't likely to change at all by moving it.  And Windows 7 takes care of fragmentation on it's own.  (Or you can use a third party tool for this.)

     

    The only scenario I'd agree that moving it is a good idea is when the boot drive is an SSD (which neither of us generally recommends anyway.)

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 19, 2011 2:19 PM   in reply to RFDPiper

    Why is everyone so down on RAID 0?  As far as losing information, if you lose the E drive in your example above, you lose all your media too.

     

    True.  But with one drive, you only worry about that one drive failing.  With a RAID 0, you have to worry about either drive failing.  So the chances of losing the data increase with every drive added to a RAID 0.

     

    As far as speed goes, you won't need it when using most standardized camera formats.  A single modern drive is plenty fast enough for even multiple streams.

     

    Where the speed is really needed is with Uncompressed or 4K formats (like form a RED camera).  But when that speed is needed, I recommend a RAID 3 using an Areca controller card.  It offers most of the speed of a RAID 0, but with built in data security in the event of a drive failure.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 23, 2011 1:34 AM   in reply to RFDPiper

    Ingesting over HD-SDI or even over the poor man's version, HDMI, requires huge amounts of bandwidth, up to 1.485 Gbps and that is only doable with raid arrays. However JBOD is not advisable and using different disks in a raid0 mens that the size of the smallest disk is all that is used for the array and the rest is not used. In your example mixing 300 and 450 GB disks in an aid0 means that your storage capacity is limited to 4 x 300 GB and is far from optimal because of differences between the two kinds of disks.

     

    For all arrays, stick to the same brand and model for all the member disks.

     

    With HDV you have the compressed tapes with the original footage, albeit in 4:2:0 colorspace, as a backup and thus you can use 2 x 450 aid0 for your media and projects and the other 2 x 300 aid0 for media cache, pagefile and exports. Even then, this may not be enough for uncompressed. I think that with your workflow a dedicated raid controller and a large number of disks in a parity raid is the better solution. Say a setup like:

     

    C: OS & programs

    D: 2 x 300 aid0 for pagefile, media cache, previews (sustained transfer rate around 220 MB/s)

    E: 5 or 6 x 1000 Raid3/5 for media, projects and export (sustained transfer rate around 330 with 5 to 410 MB/s with 6 drives)

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 23, 2011 11:43 AM   in reply to RFDPiper

    There are some exceptions, but most people, when having finished their project and start exporting think "I'm glad it is finished now. I'll tell my wife/Managing Director/client that it is finished and rendering out to Encore or whatever. I'll grab a beer while I wait for this process to finish to celebrate this milestone."

     

    Remember exporting is the single last step in a project (apart from the authoring in Encore) and happens only once in a project. You may gain a few timeslices by exporting to a different disk if you have one, but if this means you have to reduce your raid to one less disk, the penalty in performance is bigger during the editing phase than the gain during export.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 23, 2011 2:52 PM   in reply to Harm Millaard

    Thanks again Harm and everyone for some good education. 

    I've read different things about exporting and I'm a bit confused. 

    Is the CPU the bottleneck in exporting, not the speed of the drive? 

    I think I read that you could even use an external USB 2.0 drive to export to? 

    If exporting to h.264, does a RAID 0 export drive not increase the speed of the export with an i7 2600K (with either mild or no overclocking) & 16GB RAM? 

    Can the CPU not go as fast as a RAID 0 can allow?

    Thanks again for everyone's help.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 23, 2011 3:21 PM   in reply to RFDPiper

    RFDPiper,

     

    In your case, you will need multiple 1TB enterprise-class hard drives and a discrete hardware RAID card. None of the hard drives currently in your posession are suitable for parity RAIDs such as the one Harm and I recommend.

     

    My recommended setup would be as follows:

     

    C: Two 450GB 6 Gbps Velociraptors in RAID 1 (450MB effective total capacity) for the OS and programs

    D: Two 300GB 3 Gbps Velociraptors in aid0 for pagefile, media cache and previews

    E: Multiple 1TB enterprise-class drives, such as the Western Digital RE4 or Seagate Constellation series hard drives connected to a discrete hardware RAID controller card set to RAID 5 (or RAID 3 if using an Areca card)

     

    Plus, your existing 1TB WD Black and 2TB NAS drives for backups.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 23, 2011 3:23 PM   in reply to flyingfish4

    flyingfish4,

     

    (R)aid0 will not speed up the H.264 encoding at all. This is because the maximum bitrate for H.264 video content is far below even the minimum sequential transfer speed of a single hard drive. Plus, H.264 encoding performance depends almost entirely on the CPU.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 23, 2011 7:02 PM   in reply to RjL190365

    Thanks RjL.

     
    |
    Mark as:
  • Currently Being Moderated
    Jun 30, 2012 9:12 PM   in reply to RFDPiper

    That would speed things up.

     

    But why render out?  Why not use Dynamic Link?

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points