NEVER NEVER 10 or 1.
raid 0, 2 sets for avchd
back up externally
The charts on Tom's hardware website suggest that Raid 0+1 is not very efficient. The optimal speeds are acheived in Raid 0 with 4 hdd:
Of course Harm's build with 12hdd will provide stellar perfomance but it is not even in the Tom's charts
I currently have only 3 hdd but I'm planning to expand in order to make the workflow with AVCHD files really 'flow'.
I'm curious in this regard if I start building gradually Raid 0s what would be the first raid build deicated to: the media source files? the media cache? rendered files? etc.
for AVCHD anything past 2 sets of raid 0 is a pointless thing.
adding a "cache" drive also did nothing
So - RAID 0 with 2x2 disks? If I have
2X1,5 and buy two more - same size (same manufacturer???) - and put them in RAID 0 - would I get 2 disks showing up in My Computer on 1.5 TB each?
What I am concerned about is the very big risk of loosing data....And how to move things from my system (see above) - can I backlup drive C and D and just move my data files from Drive E - and do a RAID = and then copy my old drive C to new disk1(stripe?) and copy old drive D to new disk 2 (of cvoruse calling them disk C and D. My drive E is only data - no software - so I could just copy all that to new disk 2 - or is this not possible for some reason? Do I have to start all over - installing everything from the top (takes at least two days for me I guess....haven´t gpot the time for it right now...
Capture/Work drive: 2 x 1.5TB seen as 3TB total in raid 0
final output/render drive: add 2 x 1.5TB seen as 3TB total in raid 0 (or any size)
back up to an external solution.
this could be NAS, eSata or even a single 1-2TB drive. or even another computer on the network.
C drive same thing.
no need to reinstall your OS drive. NEVER raid your OS drive unless raid 5,6
rather pointless when an Ext drive is cheap.
with a program like paragon (we ship with every system) you can set it to auto back up nightly or weekly.
Just to sum up:
So I could keep my old disk 1 (wich is drive c and d containing my OS on drive C and programmes on drive D) and then buy one (or three) additional disk(s) for the Raid - consisting of my old disk2 and one (or three) new disks, right (if one - only RAID0 with 2 disks if three RAID0 with 4 disks, right?
And all software on the old disk1 - and data/footage/work etc. on the RAID?
Sorry for asking this stupid, but I really do not quite think I understand everything here...:)
Message was edited by: Moxtelling
on your C drive should be
OS and ALL programs
thats leave now 2 sets of 2 drives in raid 0.
so total 5 drives plus optical.
Let me give a simple guideline:
You need one disk for OS & programs.
This could be either a single disk, or in case of the paranoid a raid1, or ideally a raid10 for the paranoid in a hurry. Provided you have a UPS so you don't get corruption on your mirrors.
You need separate disks for editing.
If you have 1 - 3 disks use them as separate disks or 2 of them in raid0.
If you have 4 - 5 disks consider raid3 if you are in the lucky situation that you have an Areca controller, otherwise consider a raid5.
If you have 6 - 8 disks consider raid3, raid5 or raid6, possibly with a hot-spare.
If you have 8+ disks consider consider raid30, raid50 or raid60, possibly with hot-spares.
What is the downside of using few sets of 2 discs connected in raid 0?
2hdd in raid 0 for renders/previews plus miscellaneous
2hdd in raid 0 for original media, and
2hdd in raid 0 for media cache and exports
The downside is security / redundancy. A (R)aid0 is only useful to get speed, but it offers no security. By going for 6 disks in 3 raid0 arrays, you have three times a chance that one disk failure will destory all your data on an array and each array has double the chance of loss of data over a single disk.
If you were to configure these 6 disks as a single parity raid, you can lose 1 disk without losing any data at all, if you were to use raid6 or a raid3/5 with a hot spare, you can lose two disks at the same time without losing any data.
The other drawback is speed. A single large parity raid will have better performance than multiple small arrays, so in the latter case you again have to worry about distributing accesses across disks.
In your example you covered that nicely, so the only downside would be security / redundancy.
I understand. If I may, I have a relevant question:
If you have to put things in priority order, what elements of the workflow with avchd content in PrPro CS4/CS5 would benefit most of
using hard drives connected in Raid? Here are the 'elements' I have in mind: media cache and renders/previews, original media, exports.
In other words, which of those you would put first on a Raid configuration if you build your system gradually?
I understand your question and answering it is what we are here for, isn't it.
Exports would be last on my list to put on the raid. Export is a one-way street and time is not really critical.
Media cache and renders/previews is useful to put first if you do a lot of scrubbing and previewing. That leaves media in the middle.
Thank you Harm. I did not mean to hijack Morten’s thread here (probably he is presently dreaming about that bike he should have bought instead of the nVidia Quadro FX GPU) but I hope he will find this discussion helpful for his project.
I will definitely experiment with my first ‘Raid 0’ this weekend and it will be dedicated to media cache and renders/previews files as Harm suggested. The predicament I face is that my chassis (Cooler Master 690) has space only for five 3.5 form factor drives. I have three drives at the moment:
D: page file, media cache, renders/previews
E: media and exports
This leaves me with space for two more hard drives but I’m reluctant to put my media files on a Raid 0 array.
In the past it was so easy. Just capture from your tape based camera to a raid0 and if you lose your data, just capture again from tape. With card based cameras it is no longer so easy. Where do you keep your backups? You need to use those cards again.
In my PC Buying guide I advise people to go for a big tower because of the easier and better cooling, and the better expansion capabilities.
Now with CS5 and MPE, I foresee that disk I/O will be a bottleneck in a lot of cases that don't fit in the case (chassis).
Again in our benchmarks of CS5 a media "cache" drive did nothing at all for performance.
they are pointless. at least for AVCHD and Red.
Since you did tests, can you advice me what is better to upgrade first for a smoother workflow with AVCHD files in CS4: RAM vs. faster hdd (raid 0)?
Presently I have just 6GB of RAM. I specifically experience hard times during playback of PiP edits.
Morten, the bike is a beauty but lacks 'Turbo-V' which is a must for editing avchd content. I guess you'll have to paddle harder during render times
i say go for ram. 12 gig is bare minimum and with CS5.... 24
Sorry to bump into this thread, but I have a question regarding that as well.
I was also considering to get 2 HDD on a Raid 0 containing the Media cache, Preview files (and maybe page file?).
The reason for that is that Raid 0 is risky, but those files are not much of a risk loosing, and I figured perhaps it would improve performance.
Scott, when you say another disk for media cache and previews does not show performance improvement: Do you mean disks in a Raid 0? Did the benchmark test a setting like that?
From what I understood, the benchmark shows the difference between putting all the files on a fast Raid and putting all of them on Raid apart from the cache files put on a separate non raid disk.Perhaps a 2 disk array like that for media cache would be better than no Raid at all? Did I misunderstand?
Are we talking about the same benchmark? I read Bill's Benchmark:
"a RAID array for the project files. Specificfunctions like aseparate dedicated drive for writing Output files or Preview files areof less value as it may just slow things down compared to a highperformance RAID, the only way totell is run the PPBM4 benchmark."
Harm, if you could write your opinion as well, I'd appreciate it.
Your answers would be of great help!!! Thanks a lot!
probably getting into a symantics thing.
a drive for temp files (cache)... windows and Adobe, was completely pointless.
where you ingest to i call the work/capture drive. this should be raided
then the drive you do your final output to i call a render drive. this shoule be raided and seperate
so you are never writing to the drives you read from!
past that i have seen liitle point ot having any other drives aside from archival/backup
unless you are recording multi track audio
So basically what you're saying is that a separate drive for media cache + preview files would not improve performance regardless of having raid?
Could it perhaps help in the timeline playback?
So you suggest:
work/capture=media - read (raid)
render drive= project file+preview files + exports - write (raid)
In testing and those benchmarks Scott posted, I have seen zero performance increase in either export or realtime playback having a separate Media Cache/Preview files drive. Also we normally have the project directory with the source material. We also leave the Preview directory to follow the projects directory since once again there is no performance gain in realtime playback in switching the preview directory to the render drive. Switching this to custom has caused issues with video i/o cards in CS4 and I wouldn't be suprised if it did in CS5 as well. Rendering the sequence may take a little longer since it's reading and writing to the media drive when rendering but that wont be nearly as time intensive in CS5 as it was in CS4. The Exported files will still go to the render drive and that will be faster keeping the preview files on the media/source drive.
For simplicity purposes, would you indicate the allocation of hard drives you recommend (indicating their RAID config if any)? Like this:
D (2 drives raid 0): rendering, media cache
Thank you for clarifying that!
If I understood him correctly:
Drive1: Project file + Media (source material) + Preview Directory
Drive2: Exported files
Yes that is right.
Drive 1 - OS
Drive or Raid - Source Media, Project Files, Preview Directory
Drive or Raid - Export, DVD Auth Projects, AE Cache Files, Photoshop & Illistrator Scratch Files
In case you make a Raid array, could you please post the difference in performance you notice?
Just a rough rule of thumb, based on personal experience and with an Areca controller for raid3 or raid5:
Raid0: Approx. 0.9 x the number of disks in the array
Raid3: Approx. 0.85 x the number of disks in the array, minus 1 (for parity)
Raid5: Approx. 0.8 x the number of disks in the array, minus 1 (for parity)
Notice that raid0 on IHCR or Marvell chips show the same performance degradation as fill rates go up that single disks do.
Examples of sustained transfer rates:
Single disk: Around 100 MB/s
2 disk raid0: Around 0.9 x 2 x 100 = 180 MB/s
3 disk raid0: Around 0.9 x 3 x 100 = 270 MB/s
8 disk raid0: Around 0.9 x 8 x 100 = 720 MB/s
6 disk raid3: Around 0.85 x (6 - 1) x 100 = 425 MB/s
6 disk raid5: Around 0.8 x (6 - 1) x 100 = 400 MB/s
12 disk raid30: Around 0.85 x (12 - 2) x 100 = 850 MB/s (raid30 is two raid3 arays striped)
This is no guarantee you will experience the same ratios, because this was only on my system, but as an approximation, you have an idea.
I have 4 hdd in my system. I experimented yesterday with setting up two of them in Raid 0 and deticating the new array to 'scratch discs' and previews.
I'm working with AVCHD files which I edit in their native format. My style of working is to do often renders as I do colour correction and need to see the results. After setting up the Raid 0, I noticed that the renders have become faster and the previews/scrubbing flowless. Unfortunately, the preview of the avchd before/after edit (without rendering) is still choppy/stutturing and problematic even when working with a single layer. I attribute this to my low RAM (6GB). Dedicating the raid array to the original source media only - did nothing to speed up the preview of avchd content on the timeline (before render!). So, my next step is going to be to increase the RAM to 12GB. I hope this will finally resolve the problem with working/previewing avchd files on the timeline without rendering them first.
I do get yellow line when I transfer avchd files on the timeline but regardless of that their playing on both the source monitor and output minitor is stutturing and often stops. So, I cannot preview properly anything without first rendering the footage. The raid 0 for renders will speed up this process for you. For me, however, it's all the same if I will need to wait 8min (no raid) or 5.30min (raid 0) for the render to complete to see the results.My goal is to be able to preview the results without extensive rendering. I guess more RAM is the answer.
My specs are:
nVidia GTS-250 1GB Ram
Wow, this is really interesting information!
Your rendering time went down by 30% by having the scratch and preview files on a Raid 0!
I think that's significant because that could be done by most users without the risk of losing important media or spending too much money.
By the way, did you also put the media cache on the raid? (Edit>Preferences>Media>Media cache)
I hope the 12 GB RAM will help you.
Are you running Ppro CS4? If so - try going for PPro CS5 - or you will get nowhere with AVCHD.
I have had all the stuttering issues you talking about using CS4, but with PPro CS5 most of it is gone. Still I can have issues with stuttering, but if I only use different clips and accellerated effects I can have 3-5 layers running almost smoothly at the time. Scrubbing is still somewhat stuttering, but still better than with CS4. You have to use either the "Hack" described in this forum somewhere - on a GTX 260 - should work with your GTX 250 or my newly bought FX3800 (there is no difference in performance between these two cards - I work professional with video so I can not risk anything using a hack and a non-supported card - though it is quite an investment).
It might be related to your low ram as well, but even with quite a few layers and clips I don´t use more than 3-4 GB of ram...how much free ram do you have?
Have you tried excluding folders in your Antivirus software as well? You might exclude folders with your media-files and the Adobe installationfolder. That might speed things up a bit - if your antivirus scans all your big media-files constantly.
Anyway - sounds good you have had quite a gain in performance afterall. I will seriously think about getteing a RAID0 in my system as well some time - when I have payed the bill for my FX3800... ;-)
Hi Naama and Morten,
The time comparison I reported was an approximation. Defenetely the raid 0 in my case improved the render/preview but as I said did nothing to help the stutter if you opt for putting your source material there.
Morten, I'm indeed using the CS4 and hope to get my hands on CS5 soon as workig with avchd content in CS4 is a torture. Back to the RAM question. I noticed that while editing avchd in CS4 with 6 gigs of RAM the system utilizes around 3.5GB of the memory and the remaining 2-2.5GB remain free. I was wondering: if I increase the RAM to 12GB will the system increase the RAM utilizaiton proportionally and will this help me preview non-rendered avchd files without stutter? I hope the answer is affirmative as if it is not then the only hope is CS5.
Naama, I put the cache on the Raid array as I followed Harm's advice for hdd utilization with 3 hard drives (I have 4hdd buttwo of them are in raid 0) which is:
C: OS & programs
D: page file, media cache and renders/previews plus miscellaneous
E: media and exports
Morten, are you suggesting that if I go for the GTX260 and use the softmod I will experience a marked improvement in working with avchd files (under MPE) that approximates the workflow with Qudaro FX GPUs? I cannot afford a Quadro card but I can upgrade fom GTS250 to GTX260 and do the softmod.
P.S. Morten, I saw last night here in Toronto (Canada) a famous band from the '80s: a-ha (from Norway). The lead singer has the same name as you. If you can sing like him, you'll be a star soon
Hahahaha - AHA - yes - they are very known here in Denmark as well. Morten is also a common name in Norway.
I have family in Toronto - or near by - a cuisin - but I have not seen him for many years...since he was 1 year old actually - and have no contact with him today. Sad
Back to PPro.
Yes - I suggest you wait a bit with more RAM - you might need it sometime, but thats not the issue for you right now. The issue is PPro CS4. You need CS5 to gain better performance. And you might even get around with your GTX250 and the softmod. So get it ASAP.
And yes - I can confirm - I have not (yet!) seen any performance or any other difference between my old GTX260 with the softmod and my new FX3800. I feel a bit like a complete fool, but guess (and hope - and pray) it´s worth it in the long run! Adobe might make some improvements in a while that makes the officially supported cards even better compared to non-supported. But I do not know. I only have bought the FX3800 because I can not risk to have serious issues when making my living on this....
I have had serious issues editting AVCHD from my Panasonic AG-HMC-151 - as you will see in a lot of my posts in this forum. But as i have allready said - many of the issues is now histpry with CS5. But there are still some issues - among others there is an issue when you put subclips from same AVCHD masterclip on the timeline right after each other and make fx. a crossdissolve. That makes PPro go nuts and even crash. This is an issue in both CS4 and CS5. We have reported this to Adobe and got confrimation from them that they they can reproduce this issue and that it is officially a bug now - and they are looking into it to solve in in near future. Hopefully very soon because this is a workflow that is used a lot by video-editors I guess.
So - get PPro 5 and see if a lot of your issues will not dissapear...?
Oh - am I tired of writing "issue in this forum...?? At least 117 times in this post only...
Man... I read through this like 5 times and I am still a bit confused... I am gradually building here as I dont need to edit until this summer (i noted upgrades below that will be done before editing). Please note that this is just for fun... not professional. I've been use to rendering since 6.0 days but I must say, with the "soft mod" on my GTS 250, the MCE has kicked in and I have been able to achieve smooth playback with AVCHD, no rendering, using effects with my current setup (1 drive running everything too).
4 GB RAM (upgrading before editing)
4 WD Black 750GB Drives + 1TB External for backups
Nvidia GTS 250 1GB (upgrading when new list comes of certified cards)
Drive 1: OS, Programs
Drive 2-3 (raid 0): Pagefile, cache, previews
Drive 4: Media, Output
Drive 1: OS, Programs
Drive 2: Pagefile, cache, previews
Drive 3: Media
Drive 4: Output
Drive 1: OS, Programs
Drive 2: Pagefile, cache, previews
Drive 3-4 (raid 0): Media, Output
Open to suggestions.... maybe something cool that i dont even know about LOL.
Bryan and Morten,
Can you please provide me with a link how to do the softmod on GTS250? I'm not into posiiton to purchase GTX285 at the moment so, I'll give the 'soft mod' option a shot before I decide which to upgrade first - the GPU or the 6 gigs of RAM i have in my system.
My misfortunes with Adobe/AVCHD are the exact same ones experienced by Morten - even a sinle layer of avchd content in the timeline stutters in both the source monitor and the outcome monito in CS4 and is pain to edit for that reason. Try to add a single video effect (like colour correction) or even a signle transition and ... oh, boy! the editing becomes hell and I start feeling sorry that I did not go for Canon HV40 camcroder that shoots HDV on those good old casettes.
Back to your question Bryan regarding the allocation of hdd - this is the advice I got from Harm on the topic. Also included here is advice on which drives to set in Raid 0 in priority order. Hope this helps:
System with 3HDD
C: OS & programs
D: page file, media cache and renders/previews plus miscellaneous
E: media and exports
System with 4HDD
C: OS & programs
D: Windows page file and renders/previews plus miscellaneous
E: original media
F: media cache and exports
C: OS & Apps
D: pagefile, temp files (Windows), miscellaneous and exports.
F: projects and rendered/scratch files
Which to put first on Raid 0?
1. Media cache and renders/previews is useful to put first if you do a lot of scrubbing and previewing.
2. That leaves media in the middle.
3. Exports would be last on my list to put on the raid. Export is a one-way street and time is not really critical.
Sorry I should have told you I was using CS5. When I say soft mod, I meant the hack posted somewhere on here. It worked for me so far with minimal testing. When i was using CS4, I did a cineform trial and it helped a lot. After the trial ended, I didnt have enough money to purchase cineform so i haven't edited anything since. Last week I got an oppurtunity to build a new machine and I opted for some upgrades. I'm still not complete yet, but the 3 extra drives will help.
From looking at their web site, neoscene seems to cost 129$ which is much less than upgrading to CS5. I assume CS5 is more convenient cause you don't have to convert but if you're on a budget that could be perhaps a good solution.
From what I've been reading, that hack is intended for the use of the new Mercury in CS5 for graphic cards that support Cuda but are not authorized by Premiere. I don't think it will help you in CS4 or improve your graphic card's abilities.
But I might be wrong.
I almost forgot, are your HDD fast enough? Could they be slowing down your system?
I don't know if this helps anyone but perhaps it could. Disk set up could be confusing.
Here are Adobe's recommendations:
By default, scratch disk files are stored where you save the project. The scratch disk space required increases as sequences become longer or more complex. For best performance, it is recommended that you dedicate a hard disk or disks, (other than the disk where you keep your project file, your operating system files, or your files for applications) strictly to your media assets. That way your media disks can access and play media files as fast as possible, without having to access other files. In terms of performance, it’s usually best to dedicate a different disk to each asset type, but you can also specify folders on the same disk. You can specify unique scratch disk locations for each of the following types of files: Captured Video, Captured Audio, Video Previews, Audio Previews
Optimizing scratch disk performance
For optimum performance, follow these guidelines:
· If your computer has only one hard disk, consider leaving all scratch disk options at their default settings.
· Set up scratch disks on one or more separate hard disks. In Adobe Premiere Pro, it’s possible to set up each type of scratch disk to its own disk (for example, one disk for captured video and another for captured audio).
· Specify your fastest hard disks for capturing footage and storing scratch files. You can use a slower disk for audio preview files and the project file.
· Specify only disks attached toyour computer. A hard disk located on a network is usually too slow. Avoidusing removable media because Adobe Premiere Pro always requiresaccess to scratch disk files. Scratch disk files are preserved for eachproject, even when you close the project. They are reused when you reopen theproject associated with them. If scratch disk files are stored on removablemedia and the media are removed from the drive, the scratch disk won't be available to Adobe Premiere Pro.
· Although you can divide a single disk into partitions and set up partitions as scratch disks, this doesn’t improve performance because the single drive mechanism becomes a bottleneck. For best results, set up scratch disk volumes that are physically separate drives.