Disk Configurations

Roy,

I will answer your questions tomorrow. It is midnight now and I'm going to sleep.

I'm looking forward to an answer too, because I have some of the same questions. I'm currently working with a systems integrator on a quote, and we are hashing out some details about a few things.

I do a lot of uncompressed 10-bit, as well as some 1080p60 projects. So, for the RAID, to date, I'm going for an Areca ARC-1880ix-16. Funny thing is, there is not much price difference between the 12 and 16 port model. So I'm going to go with the 16-port model, and upgrade the cache to 4GB. Seems well worth it. I'll probably start out with an 8 disk RAID setup, and upgrade it n the future if need be.

We did toy with the idea to build the RAID around SSDs.... Ouf, imagine having an 8 SSD RAID! (Corsair SSD Force Series 3). But realistically, I'll most likely go with normal SATA III drives. And since they are so inexpensive, I'll probably fill it to the brim. Or most likely, the capacity that the case can handle.

But Roy has a good point. What about distributing the load on the array? Would it be more appropriate to make 2 RAID groups on the card? To balance the traffic of the media, cache, previews, pagefile and export?

-----------------

Roy, I can start answering some of your questions though (My years of being a PC tech comes in handy sometimes, hehe)

Would there be any problems having multiple RAIDs?  In the above example, the RAID 3 would require a hardware controller and the RAID 0 could run off the ICH10R on the motherboard.  Can  ICH10R support multiple RAIDS (more than one RAID 0) and can a hardware controller (say, an Areca) support more than one RAID?  If so, would it be better to run both the RAID 3 and RAID 0 in this example off the Areca?

From a technical standpoint, there are no problems running multiple RAIDs. But there would be a performance drawback if the RAID was software only (OS managed). Thankfully, on-board RAIDs do help, but there is still some CPU overhead to deal with on-boad RAID5, and very minimally RAID 0. Having a RAID card is always the better option if you can afford it. The performance, manageability and flexibility are unmatched, compared to any on-board motherboard RAID controllers. RAID 0 is simple and does not need much resources. So, yes, you could run the RAID 0 from the on-board controller of the motherboard. It would theoretically "offload" the 8x PCIe lane from extra traffic, but practically, I seriously doubt the disk I/O would exceed the PCIe bandwidth in the first place.

What about using RAID 0 for source media?  I understand the likelihood of problems increases with the number of disks but what does that mean in the real world?  I've been using my current drives (Seagate SCSIs) for about 7 years and have never had a problem.  In fact I've owned computers with hard drives since the early 80s and don't believe I've ever had a disk fail on me.  Of course everything needs to be backed up but how often might I be rebuilding a RAID 0 due to disk failure?  Maybe I've been very lucky or maybe "they don't build 'em like they used to".

I use P2 media. So I practice double copies. A working copy on the computer, the other on an external HD as a backup copy. Everyone using solid state media to record on, should do the same. Having said that. You know what RAID 0 means?  Zero chance of data recovery if 1 drive fails. The more drives in a RAID, the more likely a problem can arise. Packing drives tightly together will produce more heat if not well ventilated, and will reduce the life expectancy of any drive. I have come across some bad disks in my 20+ years dealing with computers as a tech. Not that many, but enough to not trust them, and enough to practice backups even if I had a RAID 5 or 6 (and a hotsparet). Even though I have a backup copy on an external drive for my source media, and even though I try to backup as often as I can, I can still loose other things (ancillary files) in my hypothetical RAID 0 media drive. Worst case? I could loose a day's worth of work, plus what ever time it takes to rebuild and restore everything from the previous night's backup (if I didn't forget). Time is money for most of us. And investing in a proper editing system is something I don't take lightly.

A hardware controller is required for RAID 3 and strongly recommended for RAID 5 but do they offer an advantage for RAID 0?  What about for RAID 10?  One advantage would be providing extra ports since most motherboards only provide 6 SATA ports.  Does one motherboard offer any better SATA and RAID performance than any other or are they all about the same in that regard?

There are no major advantages to use a RAID 0 or 10 on a standard addon hardware RAID controller, other then to free up ports on the motherboard, or have a higher disk count on your RAID. But higher end RAID cards with bigger cache, will be faster. On-board RAIDs do have some overhead, but for RAID0, it's not as drastic as RAID5. Motherboard SATA RAIDs, with the same chipset, for all intents and purposes, are basically the same performance. There may be small variations from one manufacturer or another, but nothing real world measurable.

Is there any advantage to external RAIDS other than convenience in moving data from one computer to another?  It seems like a controller directly on the bus would be faster than one connected externally.

Convenience, is subjectively proportional to your needs and disk quantity inside the computer casing. hehe What stops me from having more then 16 drives in my system, is the casing size for HDs and possible heat dissipation issues. I try to have a system that is self contained, and avoid using an external enclosure if I can. But regardless, the speed of internal and external ports on a RAID card is the same.

I've run across some things (don't have links handy) that indicate there may be problems with drives larger than 2 TB.  Is this just for single drives larger than 2 TB, RAIDs larger than 2 TB, or am I confused and this is not an an issue?

Not an issue with Windows 7.

Frederic

Roy,

Here it comes.

Your C, D, E and F configuration mimics partly my own setup. I have

C: [1drive] OS, programs

D: [raid0, 2 drives] Pagefile, Media Cache (on-board)

E: [raid30, 12 drives] Media, Projects, Previews (Areca)

A: [1 drive] Stock footage, Sonic Fire music

B: [1 drive] Exports

Why? Because the burst rate for the small Media Cache files is better on the raid0 than on the raid30. But the sustained transfer rate for the larger Media and Project files is way better on the raid30 (around 850 MB/s), and - not unimportant - the cleaning up after projects is very easy.

Head contention like you mention when more than 1 file is needed, needs to be seen in relation the number of drives. Say you need 8 files at the same time. Imagine two approaches with 6 disks available:

a. 1 raid0, 2 drives and 1 raid3, 4 drives or

b. 1 raid3, 6 drives.

For b it is simple, everything stored in one place, easy to organize. Access in distributed across 5 drives (one is for parity in raid3), each drive accesses 1/5-th of each file and repeats that for all the other files. Say each file is 500 MB, then each drive has to access only 100 MB per file, before moving on to the next file.

For a it is a bit more complicated, because which of those 8 files are stored where. Say you have stored 3 files on the raid0 and 5 files on the raid3. Then the raid3 with distribution across 3 drives need to access 1/3-rd of each file, so 167 MB per file and of course that takes longer per file. But more importantly it is more difficult to organize and clean up and the fill rates go up more on the smaller raids, reducing performance.

A rough estimation of the sustained transfer rate of solution a and b is 200+250 MB/s and 425 MB/s respectively.

Both ICH10R and dedicated controllers support multiple arrays. Dedicated controllers have the advantage of good rebuilding features for parity raids, array expansion, adding hot-spares and upping raid level, say from 3 to 5 to 6, etc. The more expensive cards do not only have faster IOP's more and sometimes expandable cache, but use multi-lane connectors, keeping the cable mess under control and make it easier to keep a good airflow in the system reducing heat build up. Additionally they offer external multilane connectors, using both 8087 and 8088 ports. Also the use of port multipliers is only found on the more expensive models.

I would say that comparing the ARC-1210 and the ARC-1880 is similar to comparing the nVidia GTX-260 and the GTX 580. Older architecture, lower clock speed, less memory and cores.

Scott is a favorite of using 2 raid0 in systems, but the drawback of that is that any performancce gain while editing is offset by the time it takes to make backups and restore them after a failure. I personally prefer the redundancy of parity raids, so I can exchange (with hot-swappable bays) the drive, continue working and have the controller do the rebuilding in the background or use the hot-spare till I have time to replace the failed disk.

I've had my share of disk failures in the past 25 years. 1 Seagate SCSI, 7 Seagate 7100 SATA (out of 12) and 1 Samsung F1 (out of 16), so I like redundancy.

Bill Gehrke has done tests with 8 disk raid0 arrays, using 8 SSD's, 8 147 G 15K SAS and 8 1TB SATA disks and it did not make any difference in performance. BFTB wise only SATA looks sensible. The larger cache on the SATA3 drives is always welcome, even on SATA2 connections. If you are looking for SATA3 disk, do not skip the Hitachi 7K3000 line. They no longer deserve the nickname 'deathstar'.

Raid 0 and 10 are pretty easy for on-board controllers. Dedicated controllers are not noticeably faster, but especially with raid10, remember that it takes at least 4 SATA connectors when using on-board, connectors you may need for other disks or burners.

Here is an article about the 2 TB limitation: Adobe Forums: How to access more than 2 TB of disk...

Samsung F3 disks are very fast and quiet, but they have only 32 MB cache. The F4 series have one very fast model, the 320 GB, but the other models with larger capacity are indeed eco-green and not advisable. If I were to upgrade today, I would almost certainly go for the Hitachi 7K3000 2 TB model, one that is officially certified by Areca. Good performance, attractive price, 64 MB cache and sufficient space when using a 16 disk raid30, including hot-spares. I don't know how quiet they are, but with a fully loaded system like mine (max. 19 HDD's and 2 BR burners) the noise of my fans is more than the disks together.

Sorry it took so long to answer, but I hope this helps.

19 disks + 2 optical..... WOW.... What casing do you have to fit all of those? Any new recommendations on new casings for this many drives? And what drive cages are you using to fit them all? Is the heat dissipation an issue?

Lian Li PC- A77 chassis, http://www.lian-li.com/v2/en/product/product06.php?pr_index=478&cl_index=1&sc_index=25&ss_ index=61

and Chenbro hot-swappable drive cages, http://www.chenbro.com/corporatesite/products_detail.php?sku=184

Heat is no problem, but as I alluded to, the noise of the fans to keep it all cool is much louder than the noise of the disks.

When SSD's become attractive from a BFTB perspective, I could configure this chassis without any problems to fit:

2 BR burners

18 3.5" disks (15 hot swappable)

4 SSD's (all hot swappable)

I would like to thank everyone for the very instructive comments on this post !

Usually, I would be in no position to provide any good advice. But this time i would like to quickly share my recent experience regarding Raid configuration choices and the importance of some other aspects than just performance.

As stressed by Harm (and Bill and Jim in other posts) the files organization and the clean-up after projects is not "unimportant".

I don't work with 10 bit uncompressed media, but have a multi-cameras workflow and many recordings/narrations i have to capture from different card formats, precisely organize a big amount of files before editing, then work on it for days while i keep capturing different media from different projects....

After a months of spreading my different assets across my 2 set of Raid 0, while working on them then backing them up on external Raid 1, again & again, i think i'm starting to loose my hair

I can't complain about the performance, Raid 0 is FAST, but there are way too many assets/projects files/exports to organize to keep up.

Also, the lack of redondancy is a risk i start to realize, i'm not sure that taking extra-precautions is overkill when it comes to professional projects. The price difference for a good controller and disk setup is justified by the security i could get.

Using a big 8  disks array will allow me to keep a simple organisation on "one" disk, no need to worry to much about possible failure of current projects, easier to backup  and clean-up, and leave the temps files i don't need to backup (pagefile, mediacache) on a Raid 0.

Regarding my needs, security & ease of use proved to be as much important as performance.

Only my 2 yens

I'm wondering if I really need RAID 3 or 5, or can get by with RAID 0 which would not only save the cost of a controller but require less physical drives for the same throughput.

I fully understand your concerns.We all have to follow a budget. But thing of it this way, what happens if a disk fails? Are you willing to take the time to restore everything? Plus the time it takes to go out and buy a new drive and swapping it? Will you have daily backups always on hand? Will loosing time stress you? Will your client be annoyed at delays? If you can't afford a RAID card, then do yourself a favor, use the onboard RAID controler with a minimum of 3 disks in RAID5. 3 is the minimum number for a RAID5. You'll be secure in knowing that if a drive fails, not all is lost. And it will give you ample time to swap the drive and rebuild the RAID. A three drive RAID5 is not as fast as a 2 disk RAID0, as you do have some CPU overhead to manage the parity. But if you can, get yourself a forth one to be faster. Speed is wonderful. But it's like riding a fast bike without a helmet. You'll be glad to have that helmet on if you crash!

ilHowever, I just looked up PCIe speeds and apparently it's not as fast as I thought.  If I understand correctly, an 8X PCIe card has a 4 Gbit/s rate which is only a little faster than SATA 2 and slower than SATA 3 (based on theoretical maximums of course).  Plus, eSATA(2) could transfer up to 375 MB/s and I don't think I'll need a RAID faster than that (although I may be wrong).

A single lane PCIe 2.0 is 500MB/s (Bytes, not bits) So an 8 lane RAID card's theoretical maximum throughput is 8x 500MB/s = 4GB/s (Bytes, not bits) SATA2 is 3Gb/s (375MB/s), and SATA3 is 6Gb/s (750MB/s). Mind you, no single hard drive can actually achieve this speed. It's just the theoretical limit of the interface. Hence the need to RAID drives together for speed (and security).

Roy,

Harm, I was also wondering how you housed those 19 hard drives.  Are you able to squeeze them all into the 5.25” bays (the Lian Li PC-A77F only has 12) with the Chenbro cages or do you also have cages placed elsewhere in the box?

12 slots of which 2 for the BR burners. Of the remaining 10, 9 are used for three Chenbro cages for 15 hot-swappable disks. The last one only has a single disk and could be replaced with a 4 SSD hot swappable drive cage from Addonics. The case has room for three more 3.5" disks in the rear top, which is where I have my other disks. I haven't checked whether I could replace the standard rear cage with a Lian Li 5 in 3 cage, because I don't need more space, but I guess that with some mods I could make it hold 5 disks, but the immediate concern for me would be the air exhaust from the chassis. It would be like blocking an exit.

A raid controller is like a tripod. A cheap one will last one camera, maybe less, whereas a good tripod may last you a lifetime and you will enjoy using it everyday. But of course the price is different. A $ 400 tripod is no comparison to a $ 2500 Vinten or Sachtler tripod. IMO the same applies to raid controllers. Yes they are expensive but they easily last several generations of PC's.

If you decide to go for a good raid controller, get one with sufficient ports to grow. I you only plan to use up to 4 ports now, then get at least an 8-port model. If the price difference between 8 and 12 ports is small, maybe skip the 8 port and get the 12.

You can then start with say 4 disks, and at a later time simply add disks and expand the raid for increased speed and storage space.

About enterprise disks versus normal disks, it is correct that in general enterprise disks give fewer problems than normal disks with certain brands. WD Blacks have a rather tainted reputation in that area. With my past experiences with Seagate, that is not a favorite brand of mine. A lot of people here use the Samsung F3 without problems in large arrays, even though Areca officially warns against using them. Areca officially endorses the Hitachi line that I mentioned and since their $/GB is equal to the Samsung F3, but are SATA3 disks with the larger cache, I would seriously look into those disks. Whichever brand/model you choose, keep all your data disks the same. Do not mix brands or models.

Roy,

Back in the dark ages, when I switched over from LE (using Videonics stuff) to NLE, I started out with a two port Promise card. Heck, what did I know about NLE, and thought this is OK. I then added an external 4 disk SCSI array from Videoguys and that helped a lot in performance. IIRC that was a 26 GB unit that set me back around $ 2600 at that time . $100/GB

Then on a new system I got a 4-port 3Ware card, thinking that was more than I would ever need. Used it a long time and then eventually two years ago I got an Areca ARC1680-iX-12, thinking that would be enough. Well it still is and when I start building a new system, based on the i7-3960X, I am in a quandary to keep it or upgrade to the then new ARC-2xxx cards that use PCIe-3 that are rumored to be introduced Q1/2-2012. But if I do decide to upgrade the raid controller, I will definitely go for the 24-port model and max out the cache memory. That should last me a long time.

The fault I made - in hindsight - is that I tried to be what some call 'penny-wise and pound-foolish'. I  tried to get something good for the least amount possible, which is not a bad trait, but I have been too conservative. If I had not bought the 12-port version, but the 16-port instead, I would definitely continue using it in a new machine. Now I'm doubting. Well I could still find alternative uses for this card  in one Dell server and replace the CERC controller, or use it in a new server or ...

Bill Gehrke uses an Areca ARC-1880 and I use the 1680. With my raid30 array, my sustained transfer rates max out at 850 MB/s, but with the newer card BIll has, he easily attains rates over 1000 MB/s. Mind you these figures are not with everyday arrays. They are with very large arrays, that would not apply to most people.

A Samsung F3 1 TB costs around € 41 here, a Hitachi 7K3000 2 TB costs around € 82 here, No difference IMO per GB. And yes, keep all your disks the same in brand and model, so you can add them to the raid at a later date if needed. I have the older F1 Samsungs that will be hard to get now and if one of my disks in the array fail, I have two other solitary F1's that I can use to replace the failed ones with. Of course, if that happens, I need to decide what disks will be in a new array.