1 2 3 4 Previous Next 141 Replies Latest reply: Mar 20, 2012 2:22 PM by Gf1317 Go to original post RSS
      • 40. Re: Storage rules for an editing rig. Some basics.
        321 rollem Community Member

        I have a troubling situation.  My rig once was a gaming rig with an h20 custom setup.  After I configured the loop, the remaining space allowed for 1 HDD, my OS/game/ drive.  Now, this machine is loaded with Premiere Pro CS4 and I only have the one drive to work with and I'd like to get a storage drive.

         

        my rig for references; Q9650 8g 4870 Ultra 320 SCSI 300g drive

         

        I know i'm going to need a storage drive for holding my AVCHD files (MTS) so from the get go it looks like thinking outside the case is the case i'm dealing with.   Is it a good idea to use NAS for and editing machine & how will it effect my workflow?

         

        Thanks  rollem!

         

        EDIT: tried creating paragraphs - this forum program sucks

        • 41. Re: Storage rules for an editing rig. Some basics.
          Harm Millaard Community Member
          Is it a good idea to use NAS for and editing machine & how will it effect my workflow?

           

          That depends entirely on your NAS configuration. If you have an iSCSI NAS and use teamed NIC's both on the NAS and on your workstation, you may be able to edit effectively, but I have never tried it, so I have no real life experience here. I only use the iSCSI NAS with a single NIC and use it only for backup, not for editing. With a single NIC it is just too slow. Maybe you should look at fibre channel configurations.

          • 42. Re: Storage rules for an editing rig. Some basics.
            mikeleuz

            Hi Harm,

             

            I'm looking to build a system very similar to yours, minus the large raid array.

             

            For storage, I'm looking at:

             

            System/App drive: 300GB WD VelociRaptor drive

            Editing Drive: Two 1TB Samsung Spinpoint drives in Raid 0 using onboard Marvell

            Pagefile/Scratch/Render: 1TB Samsung Spinpoint drive

             

            (all drives would be backed up to a homeserver)

             

            Is this a decent setup? I'm looking for something faster than 3 single disks, but cheaper than a full RAID solution.

             

            Rest of the hardware would be P6T Workstation, i7 920, good case and ps and cooling, 12GB corsair, gtx260 or 285.

             

            Let me know your thoughts. I need to order tonight.

             

            Thanks!

            Mike

            • 43. Re: Storage rules for an editing rig. Some basics.
              Harm Millaard Community Member

              Looks like a good disk setup. Your media drives on the Marvell will be nearly twice as fast as a single disk with very limited CPU overhead. The video cards are good as well, although the 285 is a bit pricey in comparison to an ATI 4870/4890.

              • 44. Re: Storage rules for an editing rig. Some basics.
                mikeleuz Community Member

                Thanks Harm - went ahead and pulled the trigger. I need all the horsepower I can get to handle 5D mkII files!

                 

                Thanks again!

                Mike

                • 45. Re: Storage rules for an editing rig. Some basics.
                  Nicolaj T Community Member

                  Thank you Harm for very informative guides! One question about storage:

                   

                  I've just got a new Canon Legira HF200 camera, and I need to build a new PC to edit the AVCHD files. I'm only a hobbyist, so I I'm not going for the high end systems, but I need a system that is reliable and "fast enough" for making editing AVCHD a "positive experience" ...

                   

                  On my present editing machine (3 yrs old by now) I've got 3x320 GB SATA2 (7200) drives - which I guess are the only components that might be used in my new system. I've also got a WD 1TB e-SATA for backup.

                   

                  My plans for storage on the new PC is:

                  1.- 150 GB Velicoraptor for OS/Premiere

                  2.- Media drive (AVCHD-files, music, pictures etc)

                  3.- Scrath/pagefiles/???/-drive

                  4.- A fourth array???

                   

                  (And it will be a i7 920, 12GB+ RAM, ... system)

                   

                  The basic question is:

                  As a hobbyist, how much will I profit of making no. 2 and/or no. 3 (see list above) a RAID-array (0 and/or maybe 1)? I do not think I want to pay a lot to get an expensive RAID-controller. BUT disks are not expensive, so if the performance will increase significantly by using RAID1 (2 or 3 disks), and that can be efficient managed by the MBO - I'll use such a system. I'll use my eSATA drive for backup (and maybe an internal drive for backup as well?).

                   

                  Of course - I do already have the 3x320 GB SATA2 disks - where would this be most useful, or should I rather invest in som newer and faster (7200) disks?

                   

                  I guess I'll wait a couple of months before I buy, as I still have some DV-material I have to edit, which work just fine on my previous system. No need to upgrade before I need to -as prices are decreasing and components improve fast! And I guess it is worth waiting for Windows 7 64 bit - am I right?

                   

                  Thanx

                  Nicolaj

                  • 46. Re: Storage rules for an editing rig. Some basics.
                    Harm Millaard Community Member

                    Nicolaj,

                     

                    I would initially keep it simple, since the burden of AVCHD editing is on the processor and memory, not on the disk.

                     

                    1. Velociraptor for OS & Programs

                    2. 320 G for Pagefile, scratch and final results

                    3. 320 G for media

                    4. 320 G for projects

                    5. 1 TB eSATA for backups or additional media.

                     

                    Raid1 has no speed advantage over a single disk, it just duplicates the single disk. If you run out of space in the future, you could replace a 320 G disk with a 1 TB or lager disk for you media.

                     

                    Get a very good cooler for your system, since you want to overclock the i7 for editing AVCHD and get DDR3-1333 memory, not DDR3-1066.

                    • 47. Re: Storage rules for an editing rig. Some basics.
                      Andrew M Clark Community Member

                      Harm Millaard wrote:

                       

                      How do you set up your editing machine in terms of disks for maximum performance and reliability? (SSD's are left out here.)

                       

                      This is a question that often arises and all too often one sees that initial settings are really suboptimal. These rules are intended to help you decide how to setup your disks to get the best response times. Of course the only disks in an editing machine must be 7200 RPM types or faster. No GREEN disks at all.

                       


                       

                      What do you mean by "GREEN disks"?

                      • 48. Re: Storage rules for an editing rig. Some basics.
                        Harm Millaard Community Member

                        Green disk are in marketing terms ECO friendly disks that use less energy, because their rotation speed is lower. Like a car, if you do not go over 40 MPH you use less fuel than when driving 70 MPH. Slowing down the slowest component in your computer is bad, like a snail hitting the brakes before cornering.

                        • 49. Re: Storage rules for an editing rig. Some basics.
                          Andrew M Clark Community Member

                           

                          Rule 10: If you can easily replace the data in case of disk failure (like rendered files), go ahead and use raid0, but if you want any protection against data loss, use raid 3/5/6/10/30/50. For further protection you can use hot spares, diminishing downtime and performance degradation.

                           

                           

                          Is there a reference link you could provide as to what all the different levels of RAID do?

                          • 50. Re: Storage rules for an editing rig. Some basics.
                            UlfLaursen Community Member

                            I actually changed a few 'green' discs to black caviar because of your advice, Harm, and I have been really satisfied. Thanks.

                             

                            /Ulf

                            • 51. Re: Storage rules for an editing rig. Some basics.
                              Nicolaj T Community Member

                              Hi Shaluda,

                               

                              Here you could find some good information about the different RAID-levels (click on the graphics to make the demo play):

                              http://www.raid.com/04_01_00.html

                               

                              Nicolaj

                              (and thanx Harm for you advice "a couple of posts ago"...)

                              • 52. Re: Storage rules for an editing rig. Some basics.
                                Andrew M Clark Community Member

                                Thanks for the explanation Harm. Seems like those "Green" drives would be best suited for backup type functions.

                                • 53. Re: Storage rules for an editing rig. Some basics.
                                  Andrew M Clark Community Member

                                  Thanks for the link Nicolaj; much appreciate it!!

                                  • 54. Re: Storage rules for an editing rig. Some basics.
                                    5000cent

                                    Thanks for the interesting information. Two questions:

                                     

                                    Is a seperate scratch disk for basic editing (AVCHD, little video effects) really needed? Can't you just use a disk for OS/programs and a storage disk for media, project file and preview files?

                                     

                                    Scratch disk setup in premiere offers options for captured video and video previews. Assuming captured video means raw source video, this will point to the storage disk. So a seperate scratch disk would only be used for video previews, am I correct?

                                     

                                    The WD green 1TB does go to 7200 when needed and therefore seems as fast at other storage disks. Is green really that bad?

                                     

                                    @Ulf Larsson: what difference did you notice by going from green to black?

                                    • 55. Re: Storage rules for an editing rig. Some basics.
                                      Harm Millaard Community Member
                                      Is a seperate scratch disk for basic editing (AVCHD, little video effects) really needed? Can't you just use a disk for OS/programs and a storage disk for media, project file and preview files?

                                       

                                      It is about the same as having a car with a manual gearbox. Sure you can use that car and drive wherever you like while using only the first and second gear. However, if you also use 3-rd, 4-th and 5-th gear, it will take less time and entail less risks in traffic. Same with the number of disks.

                                       

                                      So a seperate scratch disk would only be used for video previews, am I correct?

                                       

                                      No, it will be used for the media cache. the indexed, conformed and peak files and possibly for the pagefile, as well as all preview files.

                                       

                                      Mechanical disks are the slowest components in a computer. To get optimal results you need to get the last bit of performance from them. Even if what you say is true that the WD green disk can adjust their speed from 5400 to 7200, consider the enormous amount of time needed to get up to speed. That is an eternity in computer terms. Two or three seconds to stabilize at 7200 is extremely slow when disk access is measured in ms. And why? There is no discernable price difference or it can be measured in peanuts.

                                      • 56. Re: Storage rules for an editing rig. Some basics.
                                        5000cent Community Member

                                        thanks for your reply. What confuses me is that premiere does not offer scratch settings for  these categories you mentioned: media cache. the indexed, conformed and peak file. Only captured video and preview video.

                                         

                                        Assuming a 3 disk setup (OS, scratch, storage), and setting captured video to storage disk (raw footage) and preview video to scratch disk, which disk is used by premiere for what?

                                        • 57. Re: Storage rules for an editing rig. Some basics.
                                          5000cent Community Member

                                          can you please eleborate on the difference you experienced?

                                          • 58. Re: Storage rules for an editing rig. Some basics.
                                            Harm Millaard Community Member

                                            The media cache location can be set in the preferences and contains the indexed, conformed and peak files.

                                            • 59. Re: Storage rules for an editing rig. Some basics.
                                              Ulf Laursen Community Member

                                              Hi

                                               

                                              I did not have any exact problems with the green ones, I just wanted to be safe based on Harms knowledge. I use the green ones for other purposes like backup and stuff.

                                               

                                              /Ulf

                                              • 60. Re: Storage rules for an editing rig. Some basics.
                                                5000cent Community Member

                                                you say you can set media cache in preferences, but I looked and it only mentions captured audio/video and preview audio/video. but I think I understand what you mean: when you set captured video to scratch disk in those preferences, media cache is one of the temporary files that will be stored on the scratch disk, correct?

                                                 

                                                so for example, I have this setup:

                                                 

                                                disk 1: OS etc

                                                disk 2: scratch

                                                disk 3: storage with raw footage

                                                 

                                                I start a new project, I set captured audio/video and preview audio/video to scratch disk 2. And I save the project file itself to disk 3.

                                                After I import video into the project from disk 3, and start editing, all scratch files, including media cache, confirmed audio, preview files, etc will now be stored on scratch disk 2. Is this how it works?

                                                • 62. Re: Storage rules for an editing rig. Some basics.
                                                  5000cent Community Member

                                                  thanks for pointing out this media cache setting. But I am confused again.

                                                   

                                                  I opened a project on my current PC (just 2 disks, one OS and one data). Scratch disk settings under project/project settings/scratch disks is ''same as project'' for captured video/preview

                                                   

                                                  Than I looked under edit/preferences/media and the location for media cach files is a folder on my C (OS) drive: user/.../AppData/etc.

                                                   

                                                  Than I tried to locate that media cache folder on my PC to see what's in it. There is no AppData/Roaming/Adobe/etc folder on my PC

                                                   

                                                  Than I looked in the project folder on my data disk and next to project file it has these  folders:

                                                  auto-save

                                                  preview files

                                                  encoded files

                                                  media cache files

                                                   

                                                  this makes my wonder: how did I get a media cache file in my project folder on data disk while the setting in premiere points to my OS disk? And where is that folder mentioned under media cache in premiere?  And why didn't they just add media cache to scratch disk settings?

                                                  • 63. Re: Storage rules for an editing rig. Some basics.
                                                    Harm Millaard Community Member

                                                    You ticked the box in the upper right part of my screenshot.

                                                    • 64. Re: Storage rules for an editing rig. Some basics.
                                                      5000cent Community Member

                                                      no, it's unticked (I never touched it). I am starting to believe this edit/preferences/media is simply not working. The media cache setting is overruled by the scratch disk settings.

                                                      • 65. Re: Storage rules for an editing rig. Some basics.
                                                        5000cent Community Member

                                                        Hello Harm,

                                                        if you look at scratch disk settings in premiere, what is the ''captured video/audio''' setting actually used for? There is no folder on my harddisk called captured video, while there are folders called media cache and preview files made by Adobe. Maybe captured video = media cache, overruling the media cache setting in preferences, explaining why I do have a media cache folder within my project folder but not on the path mentioned under media cache setting?

                                                        • 66. Re: Storage rules for an editing rig. Some basics.
                                                          BrownChiLD03

                                                          .... i was just wondering.. all these talk about HDD performance.... why did you leave out SSD again? as per my other posts i am planning to build the BEST machine money can buy (whilst still being a somewhat practical) and SSD was my first choice dedicated for OS Drive and Scratch/temp files.. was just wondering why you left it out.. is it a bad idea?         

                                                          • 67. Re: Storage rules for an editing rig. Some basics.
                                                            Harm Millaard Community Member

                                                            At the moment I wrote this article, SSD's were too expensive and did not offer a significant speed/performance gain over a good raid setup.

                                                             

                                                            One can still wonder at the current time (october 2009) whether they will offer any significant performance gain with the new controllers and trim function in Win7 over old fashioned hard disks.

                                                             

                                                            For OS, SSD's are approaching the level where you can consider them, a single 160 GB Intel SSD for around € 520 here in comparison to a single 150 GB Velociraptor for around € 140.

                                                             

                                                            For editing the capacity is still a major bottleneck. I prefer a 12 TB raid30 with 800+ MB/s transfer rate for around € 1600 (including a top notch raid controller) over a 640 GB SSD raid0 with around 750 MB/s transfer rate for around € 2080.

                                                            • 68. Re: Storage rules for an editing rig. Some basics.
                                                              tennisboy72 Community Member

                                                              Harm,

                                                               

                                                              First of all, I'd like to congratulate you on your excellent articles about your hardware configurations.  I've just ordered a new pc and I'd have a question for you.  I would like to set up a system with one HDD for my OS and 4 HHD's in raid 0 config for my main storage with a hardware raid card.  But mostly on the net when you reed about raid, it's always a "total" raid config (with all the HDD in raid & OS also on the raid) and mosstly a software raid.

                                                              Also some scary story's about suddenly having a pc that's not bootable anymore...

                                                              So my question is how about am I to go to work ? I tought of the following scenario :

                                                               

                                                              1) set up pc with 1 HDD and install win7 ultimate 64bit on it.

                                                              2) open up pc, put hardware raid controller card & 4 HDD in it.  Also connect the 4 HDD to the hardware raid card.

                                                               

                                                              then I'm not so sure, do I have to go into the BIOS at next startup to set up the 4 HDD in raid (or is this only needed for software raid ?)

                                                              Or do I only have to config the 4 HDD with the setup software that comes with the raid card ?

                                                               

                                                              Sincerely,

                                                              Frank from Belgium

                                                              • 69. Re: Storage rules for an editing rig. Some basics.
                                                                Harm Millaard Community Member

                                                                Frank,

                                                                 

                                                                First a side remark about Win7 Pro versus Ultimate. Pro is cheaper and has a smaller footprint, do you need the BitLocker and Media capabilities? If not I suggest Pro.

                                                                 

                                                                Another side remark is that Intel withdrew the trim function update because there were numerous complaints about the hardware upgrade eating the SSD's. This will only be temporary, but be warned not to try the hardwate updates at this moment.

                                                                 

                                                                You can install the SSD and install your OS and programs. It is irrelevant when you insert your raid controller into the system, because the setup of the raid will be done during boot, before Windows starts. I would seriously look at an Areca ARC-1680iX-8 card, not cheap but the best money can buy. The iX-12 model has the added advantage of allowing more cache memory to be installed, apart from the extra ports.

                                                                 

                                                                I would not advise a 4 disk raid0. If one disk fails, you lose all your data. I would prefer a raid3 (with Areca) or raid5 with nearly all cards and build in some safety or add another disk to the array if performance is so important to you. Just the other day I had one disk fail and had to exchange it under warranty. Inserted the new disk, built a hot-spare and expanded my array, which started rebuilding immediately and I could easily continue work without losing a single byte of data. It was degraded for some time, but hardly noticeable in performance.

                                                                • 70. Re: Storage rules for an editing rig. Some basics.
                                                                  acrylucs

                                                                  I thought I'd get your thoughts on a discussion I'd been having when I found my disks being the bottleneck. I base it mostly on DB experience...

                                                                   

                                                                  I have a Raid 0 and two raid 1 partitions. One of the Raid 1's is my OS and program files, the other is storage and the raid 0 was where I did my editing. I had on the Raid 0 copies of the original files, the working and I exported to that array.

                                                                   

                                                                  When looking to improve performance, the biggest factor was sequential reading vs. Random reading. Because I was reading from the disk (inputs to the rendering) and writing to the same disk (outputs of the rendering) I was effectively 'thrashing' the needle on my hard drive and generating random I/Os. I've now moved to a system where I write to a Raid-0 during rendering, and read from Raid-1s. This generated a 20-30x improvement in disk performance (edit note: the performance gain was sequential throughput - I've gone back to the CPU being the bottleneck as opposed to the disks, so can't say the actual improvement)

                                                                   

                                                                  I don't know much about the underbelly of Adobe Premiere, but in the editing process, what are the primary read and write files

                                                                  - Source video files: Read

                                                                  - Rendered files in editing: Write and read during editing

                                                                  - Exporting: Read from source files and write export files

                                                                   

                                                                  Do I have that right?

                                                                   

                                                                  Lastly, in MSFT DBs we try and match up LUNs/logical partitions with CPU cores (quad core = max 4 LUNs) - is there any value to that in video editing?

                                                                   

                                                                  Thanks

                                                                  Mac

                                                                  • 71. Re: Storage rules for an editing rig. Some basics.
                                                                    NILYANG

                                                                    What exactly is "pagefile/scratch"?. please, can you explain me?

                                                                     

                                                                    Thank's

                                                                    • 72. Re: Storage rules for an editing rig. Some basics.
                                                                      kohns

                                                                      Just a small note about   :  Rule 1: NEVER partition a disk.

                                                                       

                                                                      For those installing Windows 7 be carefull you partition your disks before installation.

                                                                       

                                                                      On a fresh (custom) installation of Windows 7, where Windows 7 is installed onto a new hard disk with unallocated disk space (no partition or volume been defined yet), or when user attempts to create a new partition out of empty drive, the Windows 7 installer will create an additional partition with the size of 100.00 MB, and mark as System Reserved.

                                                                       

                                                                      The trick is to create a single partition beforehand leaving no room for windows 7 to create the additional partition.

                                                                       

                                                                      For more informatino check...

                                                                      http://www.mydigitallife.info/2009/08/20/hack-to-remove-100-mb-system-reserved-partition-w hen-installing-windows-7/

                                                                      • 73. Re: Storage rules for an editing rig. Some basics.
                                                                        Community Member

                                                                        Bill,

                                                                         

                                                                        Couple dumb questions...

                                                                         

                                                                        I have CS3 with 32bit XP.  I transferred 5 tapes (mini DV ) to my computer via FW and did a work print.  Now, Josh ( owner of tape) has a new 64bit vista 7 machine....  There's no way to share an external 1T fw800 hard drive is there ?  Where I load the work print from my computer to ext HD, he takes home and loads from ext HD to his machine ?

                                                                         

                                                                        If it is possible to do despite diff OS, have you heard of a good fast 1T fw800 HD recently for storage ( but fast ) ?

                                                                         

                                                                        Thanks

                                                                         

                                                                        Rod

                                                                        • 74. Re: Storage rules for an editing rig. Some basics.
                                                                          Harm Millaard Community Member

                                                                          Rod,

                                                                           

                                                                          OS version is irrelevant if all versions can R/W NTFS files. FW800 is usually only available with the use of an extra PCI card that supports FW 800. eSATA would be easier and faster. WD has a good range of external disks. LaCie and Iomega have a less than stellar reputation.

                                                                          • 75. Re: Storage rules for an editing rig. Some basics.
                                                                            the_wine_snob Community Member
                                                                            Is there a reference link you could provide as to what all the different levels of RAID do?

                                                                             

                                                                            THIS will give you the specifics of the levels of RAID.

                                                                             

                                                                            Good luck,

                                                                             

                                                                            Hunt

                                                                            • 77. Re: Storage rules for an editing rig. Some basics.
                                                                              Community Member

                                                                              Harm,

                                                                               

                                                                              Thanks !  That's great news re: NTFS ext HD on different OS..  And thanks re: suggestion on brand value.

                                                                               

                                                                              Great article on Raid too !   Copied to ms Word ( for the "book" )...spelled your name right this time

                                                                               

                                                                               

                                                                              Rod

                                                                              • 78. Re: Storage rules for an editing rig. Some basics.
                                                                                kipp_litz

                                                                                Harm,

                                                                                 

                                                                                You seem to have a lot of real-world experience in the hardware side of NLE and with that I would like to ask you a question.

                                                                                 

                                                                                I am in process of designing two identical rigs, one for home and the other for work, but would like to be able to transport my media and project files between the two systems. Of course I would like some kind of RAID for data protection as well. I am not a "professional" editor but it is one of the functions I perform at work and I often end up taking my work home.

                                                                                 

                                                                                With that in mind, do you have a recommendation for an external RAID system that would work well?

                                                                                 

                                                                                Thanks,

                                                                                 

                                                                                KippLitz

                                                                                • 79. Re: Storage rules for an editing rig. Some basics.
                                                                                  Harm Millaard Community Member

                                                                                  Several solutions, depending on your workflow, the frequency you need to transfer from A to B, the size of your projects, etc.

                                                                                   

                                                                                  1. Use a NAS like the Thecus N7700 or QNAP TS-809 Pro

                                                                                   

                                                                                  2. Use SuperMicro CSE-M35T drive cages with swappable disks and transport the disks.

                                                                                   

                                                                                  3. Use a Sonnet like external raid: http://www.sonnettech.com/product/fusiond800raid.html