Well I am happy to report after a solid day of editing that my new configuration works amazingly well comparted to my old raid 5 setup. Everything was working as advertised. I now run all my footage and scratch disks from the 4 disk raid 0 array with project files on my ssd OS Drive and the back up footage and Project files backups on the External Raid 5 Array.
The only problems I encountered was when I was trying to create a funky slideshow with JPGs I took with my 7D, I didn't scale them down so they were full 18mp resolution. After the project began to become pretty complicated it has become very unstable and keeps crashing with an Nvidia Error, saying something like my system does not meet minimum requirments. Something like that. I dug around and found others were having the same issue when trying to edit picture in their timelines. Looks like I will have to build it in AE.
The other problem was stutter on 2 random clips with a disolve added. After digging more I finally found that CS5 has a know issue with AVCHD Footage that has a dissolve between 2 parts of the same clip. So I tried the work around of making a copy of the footage in explorer and importing the copy. Then using that for the second clip in the timeline. Bingo! It worked.
Multiple layers are so far playing like butter. Lets hope it holds up as the sequences get more complicated as they get nearer to completion.
Thanks for all your help.
Yes Ulf, both the rendered files and the final destination of your exports. The thinking behind it is that on export you need access to the original media, to the media cache and media cache database files, possibly the pagefile, depending on your memory, but not often to the rendered files, because if you don't have MPE, you will use MRQ and not the preview files, and if you have MPE, you do not really care since MPE is so fast. So what you try to achieve is maximum spread across your different disks.
This post is awesome, thanks!
I have a pretty awesome system set up like the 2 drive right now, but want to go to the 3 drive. Here are my questions:
Thanks again for the great info!
Sorry but i don't speak english very weel. I need your help because I want to build a new PC for video and photo editing (SD and HD video).
My problem is for hdd configuration; I don't have problem to buy 4, 5 or 6 HDD but I don't know how many TB are necessary for the scratch/temp disk and where is important to have a RAID 0.
My idea is this:
1 x 120 GB SSD for OS/software
1 x 600 GB 10'000 RPM for Scratch / Temp / Pagefile
2 x 600 GB 10'000 RPM in RAID 0 for Media / Project
2 x 600 GB 10'000 RPM in RAID 0 for Export / Final file
What do you think about this?
Thank you and best regards
Least space requirements, but most speed requirements is for scratch, temp and pagefile.
Export/ final does not require all that speed, but may need the space. You can consider adding the single disk to the export raid in a 3 disk raid0.
I've got something odd going on with 1 of my 5 F3 HDDs
In device manager, 1 of the drives is showing up as a SCSI device and that drive is being referenced as a removable device through the oprion to "safely remove hardware and eject media". I acted on removal and it then disappears from the system until reboot.
Edit: no Raid yet. All drives are plugged into Mobo sata ports.
When i'm exporting or rendering a movie to my final video, wich HDD are working? Temp and Exporting or My Project too?
So you tell me tha scratch, temp and pagefile HDD need speed and not space.
I can buy this configuration, what you think?
1 x 128 GB SSD for OS/software
1 x 256 GB SSD for Scratch / Temp / Pagefile
2 x 600 GB 10'000 RPM in RAID 0 for Media / Project / Export / Final file
I don't need a lot of space because when my movie is finished I transfer them to my NAS with a RAID 5
Excellent guide Harm. I came up with a setup that I thought would work (and it does so far), and it turns out to be your 4-disk setup (technically 5 because my asset drive is RAID0), so you confirmed that I'm on the right track.
They don't all need to be inside your system as long as you have them linked in with esata. I am running a 4 disk raid 0 internally for scratch and a external 4 disk raid 5 connected with esata for footage storage. USB will be to slow. Not sure about firewire but my money would say it is also to slow. Esata will give you the same transfer rates as though it were internally connected with sata.
My set has been great so far.
Right, eSata or bust. USB and FireWire are fine for external backup drives. I use the LaCie d2 Quadras which have all three interfaces. The metal cases are built like granite and they run way cooler than a drive in a regular plastic case.
I am having a terrible time getting a raid 3 setup with an areca 1680ix-12...am I supposed to include the OS disk in the raid or leave it seperate? I've only gotten as far as BIOS configuration creating a raid set. It shows all of the drives I have connected, 1 150gb velociraptor, 4 1tb samsung f3's, and 1 F3 hot spare. It's telling me that I only have 600gb total space. I really cannot figure out where to go from here, any help would be greatly appreciated!
Also I cannot get WIN 7 to install as it says there are no disks available upon startup...I have no idea what I'm doing here and getting incredibly frustrated! i'm trying to follow the manual, but it is not working properly as stated in the book
Alex, I feel your pain.
Whether or not to include the OS drive in the RAID depends on what the RAID is for. However, in the context of the rest of this thread, I am going to assume you are building a large RAID for storage of footage as you work on it. In that case, keep the OS drive separate. Install the OS drive on the motherboard, and the raid drives on the areca card only. The areca card I had my tech use had a limit of 4 drives. I ended up with three drives in a RAID 3 with a hot spare also on the card. That shows up as 1.81 TB for the four drives, which is about right, considering one drive is the spare, one is for parity info, and the other two lose ~28 GB in the format. My OS drive is another 640 Gb.
I don't know the specs on your card, but these are things to watch for.
BTW my tech warned me that formatting the RAID took him nearly 20 hours, but that was necessary in order to get it to show capacity.
You keep your OS disk separate, on a mobo SATA port. That is where you install the OS & programs. Then in the Areca BIOS you configure your raid and format it, following the instructions in the manual under BIOS configuration. Press F6 during boot to enter the BIOS. After you have finished that step, under Windows go to Disk Management and format the new disk. It really is not so hard. Also look here: Adobe Forums: How to access more than 2 TB of disk...
Alex, Windows 7 gets confused and stupid sometimes when you try to install it with all of your drives hooked up. With both installations I've done it has failed when all of my hard drives are hooked up. DISCONNECT all of your drives and only leave the C drive connected (that you are installing Win 7 on), and you should not have a problem. You can leave the power cables plugged in on the extra drives, just disconnect the SATA cables.
Now, if Windows still doesn't see your C drive, hop into your BIOS and make sure you can see it there. If you can't, then Windows can't, so check your cables/connections/etc. Are you doing a RAID set for your C drive? I suggest you don't (see next paragraph), but if you are, make sure the RAID settings are correct, and make sure you have the latest BIOS from the manufacturer. You may have to use a separate PC and make a CD of the BIOS updater or however your mobo company does BIOS updates. Bottom line: you need to see your drive in the BIOS in order for Windows to see it.
I didn't read all of your posts, but I'll just throw out some suggestions... you may know all this, so disregard if you do, but I'll just do a quick brain dump here: If you have a 10K drive like a Raptor, install Win 7 and the Adobe suite on THAT, it isn't worth the headache to RAID your OS drive IMO. Zero RAID isn't that much faster than a 10K drive especially if you're using onboard RAID on your mobo, and you do NOT need to RAID1 your OS drive "for protection". Instead, once you have your OS and Adobe stuff and plugins installed and set up on your C drive, you then use Windows Backup and Restore to create an image of your C drive, so you can restore it at any point in time. Your C drive, once set up, should never change. Don't use MY Documents and all that for your files, that bloats your profile and adds unnecessary stuff to your C drive. Keep it LEAN. Store your files on some other drive. I bought a WD 1TB Caviar Black for $75 on sale, gave it the drive letter J and labelled it "Junk", and that's where all my non-editing personal files and stuff goes. Great place to put your Paging file also. And make a folder called Scratch Disks and point Premiere and the other apps to use that for scratch disks. Again, keep the C drive clean. Newegg was selling the 150GB Raptors for $99 last week, and that sale will probably repeat.
The RAID setup I saw mentioned should be for your assets, you need hyper-fast read speed when editing. Yes, the MPE and CUDA in video cards are amazing, but not limitless. So, RAID your assets drive, pop the scratch disk on a separate drive, keep the C drive immutable once it is setup, and you'll have a good system.
With my i7, 12GB ram and 470SC card I get great performance using this drive config:
300GB Raptor: C drive for Win 7 and Adobe software
Two WD 1TB RE3s in RAID0 for assets, video and audio
750GB Caviar Black for Project files and Encore files
750GB Caviar Black for Photo files (I'm a photographer and keep all of my stills on a separate drive)
1TB Caviar Black for Page file, Scratch disks and all other files that are not assets or project files, basically my junk drawer.
I do break the rules and use my PC for other stuff since it's my only PC (Microsoft Visual Studio programming, SQL Server database and Microsoft Office). That should get Harm wanting to hit me, lol! SO, what I did was add another 320GB inexpensive WD drive, and when the Microsoft installer pops up with the default "C:\Program Files(86)\software name" install path, I change the C: to the drive letter of the 320GB drive so it all installs on there. When I edit, I go into task manager and stop all the Microsoft services to free up bandwidth. I'm comfortable doing that but some may not be. A separate PC for Office/etc is highly suggested, but for now what I have is what I have so I deal with it.
OK, brain dump over. If you get some help from 5% of what I wrote I'd be happy! Not everyone may agree with how I set up my rig, but I hope the first part of this post gets you through installation pain.
Thanks for all of the helpful info guys. I finally got windows 7 installed on the raptor, had to do some reformatting to be able to get my pc to see it...caused me hours of headaches! So now I am finally ready to start installing the raid. Here's how i'm thinking of setting it up:
Raptor 150gb -C: win 7, program files
4 1 tb F3's - project files, media, etc
1 F3 for hot spare
Now I am also a photographer as well...Paulie, you put all of your photos on a seperate drive correct? Would you recommend buying an external for this? I'll be running photoshop and various other programs with the adobe production suite.
Time to start installing raid, I'll probably have more questions but hopefullly it goes smoothly!
Hi Alex, yeah, Ido have a separate drive for photos just because I had the room in my big honkin' Antec case, and because it helps my tired brain keep things organized. I do have a LaCie Quadra external drive hooked up via eSata as my backup drive so I always have two copies of everything, plus any photos that are really worth keeping I also backup onto a floppy.
Yes, I'm kidding! (I don't have enough good photos to fill a floppy, lol)... make that a Blu-Ray disc. So, recap: Video on one RAID0 set, photos on a separate drive, LaCie Quadra eSata external for backups.
SO, to answer your question, you COULD use an external as long as it's eSata (USB would be too slow for large/HDR files, would drive you batty), but you'd need another external for backups, correct? Maybe in your case it'd be better to just have a Photo section on your RAID set (I'm assuming RAID5 or 1+0 based on your 4 drives?), so you would have redundancy, therefore protected media. Is there room for a separate drive for photos? I realize you already have 5 in there now!
Here's what I would do if I were you: just make a Photos folder on your RAID set, and get your rig up and running, and get "in the groove"with your workflow. THEN in the near future, you'll have a better feel for how you mentally want your rig to function and where yopu want files stored, and then the answer will come to you on how you want to deal with your photos. Then it's just a matter of copying them to the new location, whatever that is. I'm figuring that in the first three months you aren't going to be maxing out your RAID drives, so you'll have room on there, and time to think and ponder how you want your storage mapping to go. Make sense? Over the last 15 months I changed my drives around from my original plan ten times over, but now I have it just how I like it.
BTW, Windows 7 has a great utility called Robocopy, and I use it for my backups all the time. It's a command-line utility, but you can make a quick batch file and keep that on your desktop, and just double-click it to run. Robocopy is blazing fast, and what it does is copy only what new files aren't in your backup location, unchanged files are untouched. And it also creates, on the backup drive, any new folders you might have on the source drive, on the fly. Let me know if you need more info on that (you may have backups planned already).
Thanks for the tips Paulie...I'm going to keep my photos on the raid, why not? I can't afford to lose those either! I realize now that I need a scratch disk as well, don't know why I did not think of this sooner! I'm just using an external drive via firewire currently to allow me to start messing around with premiere and make a couple edits, but I'm looking to order another 1 TB drive solely for scratch. Are there any recommendations for this drive, does it need to be 7200 rpm or is 5400 just fine?
Alex, repeat after me: "I will never use a hard drive slower than 7200rpm. I will never use a hard drive slower than 7200rpm. I will never..."
I'm glad Harm replied with a link to his thread. That was a great help for me too.
Thank you Harm, very helpful!
And Paulie..."I will never use a hard drive slower than 7200 rpm, I will never use a hard drive slower than 7200 rpm, I will never use a hard drive slower than 7200 rpm, I will never....okay i got it "
Next question, why can I not find 2gb cache memory for sale anywhere? Is it just the same as ram memory only put into a cache slot on the mobo? I'm looking to upgrade from 512 mb on the areca controller...
I'm looking to upgrade from 512 mb on the areca controller...
That is only possible with the 12+ port version. You need DDR2-533 ECC or DDR2-667 ECC memory.
Paulie, this is the cache on the Raid controller, not the mobo.
Harm, or others
Do you have any experience with the Seagate 'momentus' discs, and possible in RAID, or any comment on them?
Planning to get an Areca ARC-1210 with four 600GB Velociraptors in RAID 0 (considering R3, but I can set up daily back ups, not too worried about the data). Would there be a RAID controller better suited for this without increasing price too much (~$300)? All reviews I read point to Areca and the next best controller I see is the ARC-1231ML at $600. I have 4 other Western Digital Black 1TB's that are not in RAID currently (don't really plan to either) and an 80GB SSD as a boot drive. Thanks!
WD Blacks are 7,200RPM, Raptors are 10,000RPM. But also, Blacks are not always a good RAID choice, they have time-out circuitry that corrects for erros when standalone, but can cause RAID to drop. Hence, the WD RE3 drives, made for RAID, no timeout circuitry.
Thanks, Harm, for your expertise! I've read this post and your post on general storage setup for editing systems, and I've got some questions...
First, I've got a TON of data, much of which I could probably archive to clean things up. Here's what I've got right now, and maybe you can give me some recommendations for fixing things so it'll run better (my PPBM5 shows significant slowdown in disc-land). FWIW, all internal drives are 7200RPM SATA2.
I have a 250GB WD Enterprise Storage drive for my OS/Programs, and I think it holds the pagefile... (though I have 12GB of Ram, so I'm not sure how often it's used). This one is about 50% full.
I have a 500GB WD Enterprise Storage drive which holds all my graphics, photos, and audio, along with some other generic data files (think documents, PDFs, etc. here) This one is about 60% full.
I have a 1.0 TB WD Caviar Black for my Projects/Scratch/Previews disc. This one is about 40% full.
I have a 2TB RAID-0 (made from 2x 1.0TB WD Enterprise Storage drives) that holds all my data in whatever format it came in on (most of it is direct capture from HDV in Premiere, but there's a good mix of AVCHD, R3D, and various SD stuff, too). This one is about 95% full.
I also have a 4TB RAID-0 WD backup drive that's a little over 50% full, and maintains a complete backup of all the drives in my system, minus OS and Programs. (FW800)
Then I have a second 2TB single disc drive that I use to maintain an off-site backup of whatever projects I'm currently working on, plus important stuff like photos and graphics. This one is about 30% full. (FW800)
Then I have a third 1TB single disc drive that is a general backup drive and usually remains onsite, and is about 80% full. (USB 2.0).
Ok, so there's where things are right now. Based on what I've read, I'm thinking that I might be better off doing the following:
250GB for OS/Programs
500GB for Previews/Scratch/Pagefile
3x 1.0TB drives (no RAID) for my projects/media OR 1.0TB in RAID3/5 for projects/media (but I'll have to figure a way to archive a bunch of stuff... Bluray?)
I've got room in my system to add one more drive, and I can probably get approval for purchase of one more drive if I asked my boss--so if I got a 2TB drive, could I do a 2.0TB RAID using two of the 1.0TB drives for striping and the 2TB for parity, leaving me with an extra 1.0TB? Or should I get a 1TB drive and do a 2.0TB Raid with two drives for striping and two for parity? I'm really not up on this RAID stuff, despite reading your article on it and Wiki...
Also, could I separate my current RAID-0 and not lose the data on it?
And just FYI, I'm using onboard RAID (using the ASUS P6T6 Server mobo), and purchase of a RAID controller is out of the question right now...
I primarily edit HDV/AVCHD, but I have a little bit of RED 4k and I'm hoping to upgrade my HDV camera to the JVC GY-HM700 soon.
Thanks for your advice and expertise!
I'm not sure I understand your recommendations for four disks. "D: or E: and F: can be run in Raid0". Does that mean D: and F: or E: and F:? Or does it mean D: or E: and F:? That's not the same. Hope you understand my confusion.
I run automatically in this four disk configuration because the SSD I would like to buy (C300) has excellent reads but slow writes. So it is good for OS/Programs and bad for the page file.
Europe, Middle East and Africa