Skip navigation

Storage rules for an editing rig. Some basics.

May 3, 2009 12:19 PM

  Latest reply: Gf1317, Mar 20, 2012 2:22 PM
Replies 1 2 3 4 Previous Next
  • Currently Being Moderated
    Aug 1, 2009 3:50 PM   in reply to Harm Millaard

    I have a troubling situation.  My rig once was a gaming rig with an h20 custom setup.  After I configured the loop, the remaining space allowed for 1 HDD, my OS/game/ drive.  Now, this machine is loaded with Premiere Pro CS4 and I only have the one drive to work with and I'd like to get a storage drive.

     

    my rig for references; Q9650 8g 4870 Ultra 320 SCSI 300g drive

     

    I know i'm going to need a storage drive for holding my AVCHD files (MTS) so from the get go it looks like thinking outside the case is the case i'm dealing with.   Is it a good idea to use NAS for and editing machine & how will it effect my workflow?

     

    Thanks  rollem!

     

    EDIT: tried creating paragraphs - this forum program sucks

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 3, 2009 4:03 PM   in reply to Harm Millaard

    Hi Harm,

     

    I'm looking to build a system very similar to yours, minus the large raid array.

     

    For storage, I'm looking at:

     

    System/App drive: 300GB WD VelociRaptor drive

    Editing Drive: Two 1TB Samsung Spinpoint drives in Raid 0 using onboard Marvell

    Pagefile/Scratch/Render: 1TB Samsung Spinpoint drive

     

    (all drives would be backed up to a homeserver)

     

    Is this a decent setup? I'm looking for something faster than 3 single disks, but cheaper than a full RAID solution.

     

    Rest of the hardware would be P6T Workstation, i7 920, good case and ps and cooling, 12GB corsair, gtx260 or 285.

     

    Let me know your thoughts. I need to order tonight.

     

    Thanks!

    Mike

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 4, 2009 5:30 AM   in reply to Harm Millaard

    Thanks Harm - went ahead and pulled the trigger. I need all the horsepower I can get to handle 5D mkII files!

     

    Thanks again!

    Mike

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 11, 2009 11:39 PM   in reply to Harm Millaard

    Thank you Harm for very informative guides! One question about storage:

     

    I've just got a new Canon Legira HF200 camera, and I need to build a new PC to edit the AVCHD files. I'm only a hobbyist, so I I'm not going for the high end systems, but I need a system that is reliable and "fast enough" for making editing AVCHD a "positive experience" ...

     

    On my present editing machine (3 yrs old by now) I've got 3x320 GB SATA2 (7200) drives - which I guess are the only components that might be used in my new system. I've also got a WD 1TB e-SATA for backup.

     

    My plans for storage on the new PC is:

    1.- 150 GB Velicoraptor for OS/Premiere

    2.- Media drive (AVCHD-files, music, pictures etc)

    3.- Scrath/pagefiles/???/-drive

    4.- A fourth array???

     

    (And it will be a i7 920, 12GB+ RAM, ... system)

     

    The basic question is:

    As a hobbyist, how much will I profit of making no. 2 and/or no. 3 (see list above) a RAID-array (0 and/or maybe 1)? I do not think I want to pay a lot to get an expensive RAID-controller. BUT disks are not expensive, so if the performance will increase significantly by using RAID1 (2 or 3 disks), and that can be efficient managed by the MBO - I'll use such a system. I'll use my eSATA drive for backup (and maybe an internal drive for backup as well?).

     

    Of course - I do already have the 3x320 GB SATA2 disks - where would this be most useful, or should I rather invest in som newer and faster (7200) disks?

     

    I guess I'll wait a couple of months before I buy, as I still have some DV-material I have to edit, which work just fine on my previous system. No need to upgrade before I need to -as prices are decreasing and components improve fast! And I guess it is worth waiting for Windows 7 64 bit - am I right?

     

    Thanx

    Nicolaj

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 26, 2009 4:24 PM   in reply to Harm Millaard

    Harm Millaard wrote:

     

    How do you set up your editing machine in terms of disks for maximum performance and reliability? (SSD's are left out here.)

     

    This is a question that often arises and all too often one sees that initial settings are really suboptimal. These rules are intended to help you decide how to setup your disks to get the best response times. Of course the only disks in an editing machine must be 7200 RPM types or faster. No GREEN disks at all.

     


     

    What do you mean by "GREEN disks"?

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 26, 2009 5:56 PM   in reply to Harm Millaard

     

    Rule 10: If you can easily replace the data in case of disk failure (like rendered files), go ahead and use raid0, but if you want any protection against data loss, use raid 3/5/6/10/30/50. For further protection you can use hot spares, diminishing downtime and performance degradation.

     

     

    Is there a reference link you could provide as to what all the different levels of RAID do?

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 26, 2009 8:21 PM   in reply to Shaluda

    I actually changed a few 'green' discs to black caviar because of your advice, Harm, and I have been really satisfied. Thanks.

     

    /Ulf

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 26, 2009 11:03 PM   in reply to Shaluda

    Hi Shaluda,

     

    Here you could find some good information about the different RAID-levels (click on the graphics to make the demo play):

    http://www.raid.com/04_01_00.html

     

    Nicolaj

    (and thanx Harm for you advice "a couple of posts ago"...)

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 27, 2009 2:19 PM   in reply to Harm Millaard

    Thanks for the explanation Harm. Seems like those "Green" drives would be best suited for backup type functions.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 27, 2009 2:20 PM   in reply to Nicolaj T

    Thanks for the link Nicolaj; much appreciate it!!

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 9, 2009 9:57 AM   in reply to Harm Millaard

    Thanks for the interesting information. Two questions:

     

    Is a seperate scratch disk for basic editing (AVCHD, little video effects) really needed? Can't you just use a disk for OS/programs and a storage disk for media, project file and preview files?

     

    Scratch disk setup in premiere offers options for captured video and video previews. Assuming captured video means raw source video, this will point to the storage disk. So a seperate scratch disk would only be used for video previews, am I correct?

     

    The WD green 1TB does go to 7200 when needed and therefore seems as fast at other storage disks. Is green really that bad?

     

    @Ulf Larsson: what difference did you notice by going from green to black?

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 9, 2009 4:27 PM   in reply to Harm Millaard

    thanks for your reply. What confuses me is that premiere does not offer scratch settings for  these categories you mentioned: media cache. the indexed, conformed and peak file. Only captured video and preview video.

     

    Assuming a 3 disk setup (OS, scratch, storage), and setting captured video to storage disk (raw footage) and preview video to scratch disk, which disk is used by premiere for what?

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 9, 2009 4:26 PM   in reply to UlfLaursen

    can you please eleborate on the difference you experienced?

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 9, 2009 10:34 PM   in reply to Harm Millaard

    Hi

     

    I did not have any exact problems with the green ones, I just wanted to be safe based on Harms knowledge. I use the green ones for other purposes like backup and stuff.

     

    /Ulf

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 10, 2009 7:24 AM   in reply to Harm Millaard

    you say you can set media cache in preferences, but I looked and it only mentions captured audio/video and preview audio/video. but I think I understand what you mean: when you set captured video to scratch disk in those preferences, media cache is one of the temporary files that will be stored on the scratch disk, correct?

     

    so for example, I have this setup:

     

    disk 1: OS etc

    disk 2: scratch

    disk 3: storage with raw footage

     

    I start a new project, I set captured audio/video and preview audio/video to scratch disk 2. And I save the project file itself to disk 3.

    After I import video into the project from disk 3, and start editing, all scratch files, including media cache, confirmed audio, preview files, etc will now be stored on scratch disk 2. Is this how it works?

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 10, 2009 10:36 AM   in reply to Harm Millaard

    thanks for pointing out this media cache setting. But I am confused again.

     

    I opened a project on my current PC (just 2 disks, one OS and one data). Scratch disk settings under project/project settings/scratch disks is ''same as project'' for captured video/preview

     

    Than I looked under edit/preferences/media and the location for media cach files is a folder on my C (OS) drive: user/.../AppData/etc.

     

    Than I tried to locate that media cache folder on my PC to see what's in it. There is no AppData/Roaming/Adobe/etc folder on my PC

     

    Than I looked in the project folder on my data disk and next to project file it has these  folders:

    auto-save

    preview files

    encoded files

    media cache files

     

    this makes my wonder: how did I get a media cache file in my project folder on data disk while the setting in premiere points to my OS disk? And where is that folder mentioned under media cache in premiere?  And why didn't they just add media cache to scratch disk settings?

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 10, 2009 1:50 PM   in reply to Harm Millaard

    no, it's unticked (I never touched it). I am starting to believe this edit/preferences/media is simply not working. The media cache setting is overruled by the scratch disk settings.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 11, 2009 6:54 AM   in reply to Harm Millaard

    Hello Harm,

    if you look at scratch disk settings in premiere, what is the ''captured video/audio''' setting actually used for? There is no folder on my harddisk called captured video, while there are folders called media cache and preview files made by Adobe. Maybe captured video = media cache, overruling the media cache setting in preferences, explaining why I do have a media cache folder within my project folder but not on the path mentioned under media cache setting?

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 27, 2009 9:57 AM   in reply to Harm Millaard

    .... i was just wondering.. all these talk about HDD performance.... why did you leave out SSD again? as per my other posts i am planning to build the BEST machine money can buy (whilst still being a somewhat practical) and SSD was my first choice dedicated for OS Drive and Scratch/temp files.. was just wondering why you left it out.. is it a bad idea?         

     
    |
    Mark as:
  • Currently Being Moderated
    Calculating status...
    Oct 29, 2009 5:27 AM   in reply to Harm Millaard

    Harm,

     

    First of all, I'd like to congratulate you on your excellent articles about your hardware configurations.  I've just ordered a new pc and I'd have a question for you.  I would like to set up a system with one HDD for my OS and 4 HHD's in raid 0 config for my main storage with a hardware raid card.  But mostly on the net when you reed about raid, it's always a "total" raid config (with all the HDD in raid & OS also on the raid) and mosstly a software raid.

    Also some scary story's about suddenly having a pc that's not bootable anymore...

    So my question is how about am I to go to work ? I tought of the following scenario :

     

    1) set up pc with 1 HDD and install win7 ultimate 64bit on it.

    2) open up pc, put hardware raid controller card & 4 HDD in it.  Also connect the 4 HDD to the hardware raid card.

     

    then I'm not so sure, do I have to go into the BIOS at next startup to set up the 4 HDD in raid (or is this only needed for software raid ?)

    Or do I only have to config the 4 HDD with the setup software that comes with the raid card ?

     

    Sincerely,

    Frank from Belgium

     
    |
    Mark as:
  • Currently Being Moderated
    Dec 3, 2009 7:47 AM   in reply to Harm Millaard

    I thought I'd get your thoughts on a discussion I'd been having when I found my disks being the bottleneck. I base it mostly on DB experience...

     

    I have a Raid 0 and two raid 1 partitions. One of the Raid 1's is my OS and program files, the other is storage and the raid 0 was where I did my editing. I had on the Raid 0 copies of the original files, the working and I exported to that array.

     

    When looking to improve performance, the biggest factor was sequential reading vs. Random reading. Because I was reading from the disk (inputs to the rendering) and writing to the same disk (outputs of the rendering) I was effectively 'thrashing' the needle on my hard drive and generating random I/Os. I've now moved to a system where I write to a Raid-0 during rendering, and read from Raid-1s. This generated a 20-30x improvement in disk performance (edit note: the performance gain was sequential throughput - I've gone back to the CPU being the bottleneck as opposed to the disks, so can't say the actual improvement)

     

    I don't know much about the underbelly of Adobe Premiere, but in the editing process, what are the primary read and write files

    - Source video files: Read

    - Rendered files in editing: Write and read during editing

    - Exporting: Read from source files and write export files

     

    Do I have that right?

     

    Lastly, in MSFT DBs we try and match up LUNs/logical partitions with CPU cores (quad core = max 4 LUNs) - is there any value to that in video editing?

     

    Thanks

    Mac

     
    |
    Mark as:
  • Currently Being Moderated
    Dec 26, 2009 7:00 AM   in reply to Harm Millaard

    What exactly is "pagefile/scratch"?. please, can you explain me?

     

    Thank's

     
    |
    Mark as:
  • Currently Being Moderated
    Dec 30, 2009 1:53 PM   in reply to Harm Millaard

    Just a small note about   :  Rule 1: NEVER partition a disk.

     

    For those installing Windows 7 be carefull you partition your disks before installation.

     

    On a fresh (custom) installation of Windows 7, where Windows 7 is installed onto a new hard disk with unallocated disk space (no partition or volume been defined yet), or when user attempts to create a new partition out of empty drive, the Windows 7 installer will create an additional partition with the size of 100.00 MB, and mark as System Reserved.

     

    The trick is to create a single partition beforehand leaving no room for windows 7 to create the additional partition.

     

    For more informatino check...

    http://www.mydigitallife.info/2009/08/20/hack-to-remove-100-mb-system- reserved-partition-when-installing-windows-7/

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 7, 2010 10:43 AM   in reply to Bill Hunt

    Bill,

     

    Couple dumb questions...

     

    I have CS3 with 32bit XP.  I transferred 5 tapes (mini DV ) to my computer via FW and did a work print.  Now, Josh ( owner of tape) has a new 64bit vista 7 machine....  There's no way to share an external 1T fw800 hard drive is there ?  Where I load the work print from my computer to ext HD, he takes home and loads from ext HD to his machine ?

     

    If it is possible to do despite diff OS, have you heard of a good fast 1T fw800 HD recently for storage ( but fast ) ?

     

    Thanks

     

    Rod

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 7, 2010 9:23 PM   in reply to Shaluda
    Is there a reference link you could provide as to what all the different levels of RAID do?

     

    THIS will give you the specifics of the levels of RAID.

     

    Good luck,

     

    Hunt

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 8, 2010 8:46 AM   in reply to Harm Millaard

    Harm,

     

    Thanks !  That's great news re: NTFS ext HD on different OS..  And thanks re: suggestion on brand value.

     

    Great article on Raid too !   Copied to ms Word ( for the "book" )...spelled your name right this time

     

    Rod

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 29, 2010 6:38 AM   in reply to Harm Millaard

    Harm,

     

    You seem to have a lot of real-world experience in the hardware side of NLE and with that I would like to ask you a question.

     

    I am in process of designing two identical rigs, one for home and the other for work, but would like to be able to transport my media and project files between the two systems. Of course I would like some kind of RAID for data protection as well. I am not a "professional" editor but it is one of the functions I perform at work and I often end up taking my work home.

     

    With that in mind, do you have a recommendation for an external RAID system that would work well?

     

    Thanks,

     

    KippLitz

     
    |
    Mark as:
Actions

More Like This

  • Retrieving data ...

Bookmarked By (3)