This content has been marked as final. Show 23 replies
>Or maybe I am just over-thinking this and performances differences will be insignificant.
That'd be my guess.
If it were my system, I'd use one drive for Projects, one for Scratch, one for Exports, one for DVD Images (.iso files), and set the remaining three up in a RAID 3 config using an Areca controller for Media.
Interesting. Project, Media and Program on separate drives to avoid conflicts, makes sense. When you say Scratch (as the official Adobe material doesn't really define this and there isn't a selection for it; they seem to refer to ALL of the disks as scratch) do you mean things like media cache and preview files? Also makes sense.
Having the .iso separate from export: wouldn't you say they you'd never really be exporting at the same time you might be authoring a DVD or BD from the iso, so no conflict is likely? Curious about this.
Thanks for the input. Not trying to argue; just trying to understand disk theory. This could be one of those angels-dancing-on-the-head-of-a-pin things.
>When you say Scratch do you mean things like media cache and preview files?
>Having the .iso separate from export:
I put the .iso files on a separate disk as a Master backup. One Master copy goes onto actual DVD, one gets stored as an .iso file. When my .iso disk get's full, I offload to an external and put that on a shelf, arranged by year.
Makes sense. And in thinking about it, I suppose that when making the iso file to burn to disk, you would in fact be pulling from the export you previously made to make the iso, so you would in fact be reading from and writing to the same disk at once.
And thanks for mentioning RAID3. I did have my disks set up that way at one point and that remains a viable option as well.
In the old days with Avid, we'd have one drive for Video and one for Audio. Seems that the Media cache and the preview files are all video and play in place of the original video files, so no need to put them on a separate disc. Captured audio never gets played - it's all conformed, so one drive for conformed audio only since much of the audio is embedded in our video clips anyway.
Output renders can go anywhere - the only way that would effect playback is by taking up space on a drive and causing fragmentation of files for playback (previews or cache or new captures.)
So 5 drives for Video and 2 for Audio and get an external drive for Output Renders - unless you need to edit with them in which case they'd go on the video drive. And then just defrag as needed.
I have a totally different approach - but I don't work with true HD or uncompressed SD footage and don't need the ultimate throughput. I do DV, HDV, AVCHD, DVCProHD and a little XDCAM-EX and I just put the project and all the files on one external SATAII drive in a SATAII Dock. Makes it really easy to archive. Just pop out the drive and put in on the shelf or hand it to the client. Can also switch systems fairly easily (it will have to make new Media cache database entries.)
I have no test results to give advise. Just haven't had the time and the disks to try it out, especially with 4 different OS's to try. I don't think it will make a large difference, unless - as Jim suggested - one of the single disks is getting pretty filled up.
The problem is that there are different needs for Adobe in terms of raid configuration. Rendered preview files are typically rather small and numerous, original media are typically large. Logic would indicate that for preview and scratch files raid3 could be nice with the byte level striping, but for media raid 5 would likely be better, safety would lean towards raid50. Project files don't need the ultimate in speed and that certainly applies to exports.
If you want to test it out, I would start using Vista 64 as the basic OS, test for the best results and then just use that for the other OS's. It seems like a waste of time to test each OS individually and highly impractical, since if the results differ between OS's what will you do with the data on the disks when you need to change raid levels? And the time it takes.
I would try four scenarios, but add an 8-th disk first:
1. All disks in a raid5
2. All disks in a raid50
3. 5 disks in a raid5 for all media and 3 disks in a raid3 for projects and scratch
4. Jim's approach with individual disks
My guess is that it will not make much difference. What may make a difference is the latest Rom update from Areca and increasing your on-board cache. BTW, did you exchange the Areca for the 1680ix-12? If so, you may profit from the 4 GB cache capability.
Adam, I know you wanted to keep the topic specifically to disk setup, but one has to wonder exactly what kind of throughput you are looking for and what kind of editing you will be doing.
I have worked in numerous production houses, and each one always had a specific machine for specific editing tasks. Are you looking to capture, or edit uncompressed HD? Are you looking to work with massively large files, aka, video that has been created from one continuous recording? Are you doing an extensive amount of encoding work? Or are you simply working with SD material?
The concensus has been in the past, as Jim pointed out, to keep your OS on one disk, your projects on another, your preview/scratch files on another, and finally your raw video/audio files on yet another seperate disk. Now, I have no idea exactly how much this actually improves working performance, but theoretically it should definitely help.
Yes, I probably should have gone a little bit in to the type of video I mostly do, but I thought I should keep the experiment generic and theoretical. Nonetheless, I do mainly two types of projects, always in HDV with Cineform Aspect HD:
--Multicam edits of stage productions, which usually results in four one-hour long continuous clips edited in multicam mode. Very little effects applied; maybe a few titles and cross-dissolves, but that's about it. Some color correction is sometimes necessary. I generally break each act into its own project, and each scene gets its own sequence.
--Sports videos, which usually have close to a thousand clips (!) in the project panel and end up using perhaps 500 or so, very short shots, averaging about a second or two each. Some sequences involve multiple effects on each clip. These may be broken up into several sequences, each of which sometimes gets its own project. As you can imagine, with this many clips projects are pretty slow to load, even on a fast system, and as the sequences get more complex Premiere bogs down a lot. One five minute sequence consisting of a hundred or so shots, with motion, speed, opacity or color correction applied to every clip (sometimes more than one effect) takes about 90 minutes just to render a preview. That is, when it doesn't crash. I nearly always get an out-of-memory or other unexplained error and have to shut down, reboot, and render a section at a time. CFHD-AVI exports work fine, no crashing, but are time-consuming.
Harm, good suggestions. Right now I've got all 8 internal slots filled so adding an 8th disk may not be really practical. I probably should have reiterated that I have the system drive on channel 0, and the 7 workdrives on channels 1-7. I do have an external USB drive, but I use that only for archive and backup. Experimentally, my plan is pretty much what you outlined, except in reverse -- test the four OSes with my current config (which is all one big drive, but with spanned volumes due to the 2TB limitation in 32-bit OS) and then, based on the fastest OS, kill all the others and spend some time with different disk configs. A middle step will be to add 16 extra gigs of RAM for the 64-bit OSes after testing all 4 with my current 4 gigs, just to try to tease out any differences of OS alone vs. just the presence of extra RAM.
These are all great ideas. As before, it seems there are as many ways to do this as there are users. Sometimes there is just too much choice.
Thanks to all who've posted ideas -- keep them coming. Once I get all the OSes installed and figure out licensing/activation issues, I'll post results.
Use the floppy cage for your boot disk. Use one of the on-board SATA connectors for your boot disk and still have two 5.25 slots for CD/DVD drives. Use all 8 removable slots for your storage disks. So disconnect the boot disk from the backplane and use an on-board connection. Then you have all your work disks on the backplane and the Areca. In terms of bandwidth it may even prove to be beneficial since you spread the load across the PCI-e bus and the Northbridge chip bus.
Keep in mind when testing with the 4 OS's and with your current limitation of 2 TB arrays, you will not be testing under optimal conditions. Remember, we discussed this earlier, with a 64 bit OS you will be rid of the 2 TB limitation, so with scenario 1 you could test a 7 TB array, with scenario 2 a 6 TB array, with scenario 3 a 4 plus 2 TB array. Again, I have not tested it, but I would be curious to know if there is any measurable overhead of 2 of more arrays (one 5 and one 3) in comparison to a raid50. If you are going to test it, please keep me informed.
We'll keep in touch.
Yes, this scenario makes sense, and I think it's one we discussed about a year ago. At the moment I like having the floppy for the ability to create emergency rescue disks and the like, given the problems I've had with this box. I may do this in the future.
Thanks again for all the help. PM me if you want all the gory details of the upgrade to Vista 64, now in progress...
>but for media raid 5 would likely be better
You've got that backwards. RAID 5 offers superior performance for many smaller requests (server). RAID 3 shines with bigger files and fewer requests (media).
I've got 2 RAID 0 arrays, and I back them up to a RAID 5.
For reference, backup of my system SATA drive to another single SATA drive occurs at speeds between 60 and 70 MB/sec.
Backup of my RAID 0s to my RAID 5 occurs at 23 MB/sec. I wouldn't recommend a RAID 5 for media (as Jim has already pointed out).
Jeff, are you using on-board raid5 or do you use a dedicated raid controller?
It's a dedicated RAID controller.
Not an Areca?
No. My vendor was out of the Arecas that were anywhere close to reasonably priced. I didn't need 16 or 24 ports, so I went with an Intel SRCSATAWB. Even that was almost 400 bucks for 8 ports. My plan was for Media and Scratch Disks each to have their own 2-disk RAID 0, with a 3-disk RAID5 for backup.
I haven't benchmarked my RAID5 read/write with other cards or other RAID configurations because I do only use it for backup. Should I expect better performance out of a RAID5? Should I care?
> Should I expect better performance out of a RAID5? Should I care?
For backup? No, I wouldnt care.
Comparing to a single disk; My guess would be 3 disk raid 5 will make the read a bit faster and writing a bit slower.
But, Im sure HARM has stats to answer more precisely.
Thanks for the chart.
Thanks, Harm. My IBM uses an apparently middle-of-the-road LSI chip (unless the LSI chips have gotten better since that chart was published).
>Comparing to a single disk; My guess would be 3 disk raid 5 will make the read a bit faster and writing a bit slower.
That's probably good, because if I need to recover I'd want that to be faster than making the backup in the first place. :)
Sorry to interrupt, but I tested putting the project files on a disk other than the boot drive, cache and media but didn't see any real performance gain at all.
My tests weren't scientific at all but they were big projects with hundreds of clips and I noticed nothing.
Try reading 5 HD streams (that dont need to render) at once from an IDE drive vs a raid 0 (for example). You should see a big difference.
Note; If you're using something like AVCHD footage (worst case example) you wont see a diff because the bottleneck in that case is CPU decoding the frames.
Wow, wouldn't it be funny if all this figuring out were for nothing?
Nonetheless, my experiments continue....
(Very cool charts, Harm. Links to originals? I see some drop-downs...)