[Chris Gordon]

Can you describe your setup a little bit more? What hardware and software are you using? What is the configuration?

[Chris Gordon]

I would reformat the HFS+ partition (the 900 GB one). I’m guessing something got messed up with the file system at some point. If that doesn’t solve it, repartition and reformat the entire disk.

Have you been able to test this disk on another machine? With a different firewire cable?

[Chris Gordon]

I assume that by “SAN” you mean the entire storage system, not just the Fibre Channel network used to connect the server and storage. I also assume that you run your databases on large enterprise class storage — EMC, NetApp, Hitachi, etc. In that case, look at the various storage vendor’s training programs or work with the storage admins at your current place of employment and see if you could work with/for them as an apprentice and learn that way. Different arrays from different vendors (even arrays from the same vendor) are very different and take some specialized training (formal or informal) to learn. It’s not unlike a database Just because you may be an expert on Oracle doesn’t mean you’re an expert on every database that exists. This is probably at least a 1 – 2 year endeavor.

[Chris Gordon]

Did you install, configure and start the iSCSI target software on your Linux server?

[Chris Gordon]

First big question to ask is do you need all of this data online and available to all of your editing stations all of the time? If you don’t, then you could get a less expensive and more manageable solution at a much lower price point.

I’ll assume you do need/want everything on line and accessible at once. To better frame the problem, let’s look at the number of disks you’re talking about. This is just some ballpark guesswork, but it will help give you an idea.
– I’ll assume (there I go again) we are using 1 TB disks
– 8+2 RAID6 groups (10 disks total in group, but you effectively have 8 disks for data, the other 2 are consumed by parity)
– 1 hot spare disk for every 4 RAID groups (this allows for rebuilds of failed disks onto a disk already in the system instead of waiting for you to get a new disks and replace it.)
– We only fill the disks to 75% full. Performance of disks tend to drop off significantly as you get closer to capacity. (This is different than keeping free space in a file system.)
– If I’ve done my math right, that comes out to about 205 disks to get you 100 TB of usable space (less file system overhead). This will probably end up being at least 1 42U rack full of disks.

Now that we have some ballpark of the number of disks we’re talking about, some additional things to think about.
– Do you plan to back this up? If so how? Traditional backups (a machine connected to the storage and a tape library and writing all of the data to tape) can put a hit on your storage and possibly affect performance as it sucks up all of the IO capacity on your array. Some ways around this are:
— Do your backups when no one is using the array. With 100TB you’ll need to break up what gets backed up on which days
— Make array based copies of the data and backup the copies. This will increase the number of disks you need since. Additionally, you’ll need the horsepower in the array to make the copies.
— Replicate the data to another array, preferably in another location. Mid and high-end arrays have the ability to replicate data between different arrays both synchronously and asynchronously. This can get your data to another location with little thought and forgo the need to always use tape. Or you can use the remote array for backups so your primary array isn’t affected
— I could come up with others but would need to understand your workflow more
– How much time and energy will be invested in managing all of this? There is a balance between the size/number of arrays you have and the effort to manage it all. A bunch of small 16 to 24 disks arrays with an expansion chassis or two on each may keep your price down but can result in a lot more work to manage as you have that many more arrays to deal with. Bigger arrays more cost a lot more but you only need one of them making management much easier. As you said before, people time and skills cost money. Regardless, at 200+ disks you are going to need to plan for resources (people) to take care of it on a routine basis — either your existing IT staff or contractors/consultants.
– Do the controllers in the array really have the horse power to keep up with what you want your array to do. I’d be very skeptical of something like a little XScale/ARM processor keeping up with 200+ disks in a single array, especially if you do any array based copies or replication.

All of that said, I’d go back and really look at what you need to have online on your “performance” array and what can you have in a slower/cheaper “archive” array. Needing to have 100TB+ of storage online and ready to edit is going to be expensive, maybe not really what you need and more complex than you’d like.

Like everyone else has said, talk to multiple vendors and get an idea of the solution they propose. If you really do need storage this big, I’d expect then vendors to put together proposals/bids describing their solution — and I’d expect that to be free (that’s what it costs them to get the sale). I deal with designing a lot of infrastructure solutions at work and have vendors propose solutions all the time (them showing up on site with loads of people to better understand the problem and our needs) and I’ve never been asked to pay for that. Now on going support after the purchase, help installing systems, on-going professional services, etc — that’s all something I expect to pay for, but not a proposal on what they want to sell me.

[Chris Gordon]

Before I throw out any suggestions, what is your budget?

[Chris Gordon]

First, RAID5 and RAID6 use striping and are “striping with parity” and “striping with double parity” respectively (see Wikipedia’s article on RAID for more discussion). Striping effectively spreads the IO across multiple disks giving you overall increased performance. I’m not sure what was meant by “In the world of Video Editing and Real Time RAID performance, stripping tends to be very bad” — without it, you’d be limited to a single disk.

So let’s assume a situation where you have 10 disks to work with.

RAID0, a simple stripe across all 10 disks with no redundancy. All 10 disks are used to read and write. This gives you the absolute best performance for a given set of disks, but you’re SOL when just one of those disks dies.

RAID5, striping with parity. This effectively gives you 9 disks for reads and writes with 1 used for the parity (it’s more complicated than this since the parity is distributed across the disks, but good enough for discussion). This is good for video work since video is typically dominated by simple, large, sequential reads and writes. With the reads you get 9 disks to perform the large reads. The number of disks to absorb the writes outweighs the impact of having to perform the parity calculations and do the additional write of the parity data. Now for the problems with RAID5 (or RAID6):
– When a disk dies, ALL of the other disks are impacted as they ALL must be read from in order to rebuild everything on a new disk. In addition to the hit from the rebuild IO work, any reads from the disk during the degraded time require parity calculations to give you the complete data. You should make sure your array can handle the your IO demand during a rebuild/failure situation.
– Since the rebuild so heavily exercises the remaining disks in the array, you can see additional failures occur at this time.
– Updates to files, especially when the update is smaller than the strip size, are typically slow. To do the update, you have to read all of the rest of the stripe, perform a new parity calculation, then write the updated data and new parity. Video work typically doesn’t see a great deal of updates, so you probably won’t encounter this.
– Small IOs may not get the benefit of hitting all 9 disks and their write performance may be adversely impacted by the parity calcs and writes.

RAID10, striping across mirror sets. With this you mirror pairs of disks and then build a stripe across them. You effectively have 5 disks for read and write. For simple, large, sequential reads and writes, 5 is less than 9, so you don’t get as much throughput as in the RAID5 case. At the same time, a disk failure only requires one disk to be read from with the rest of the array is unaffected (and there are no parity calcs to do a read during the degraded period, nor parity calcs to rebuild the replacement disk). Additionally small random IOs or updates do not feel any penalties from the parity calcs as there are none.

Now assuming a RAID5 of 10 disks, if I build a RAID10 set of 18 disks I would get at least as good performance as the 10 disk RAID5 and not suffer any of the problems with parity RAIDs. Of course this would cost a little bit more. Of course an 18 disks RAID0 would beat them all in performance (but may result in a resume updating event when just one disk dies).

Hope that helps.

[Chris Gordon]

Thanks. I normally work in the world of big databases where the impact of the rebuild is often too great (affects response times too much). A big thing that is starting to change this for us is storage tiering where the array has a mix of Flash and SATA drives (and maybe some FC or SAS depending on the specific solution) and the array moves data between the two automatically. The basic idea is the blocks you hit a lot migrate toward Flash and the stuff you never really touch ends up on SATA. Then putting the SATA disks in a RAID5 (or 6) is fine since you don’t feel the impact of the rebuilds so much — its on data you don’t really ever hit. Great for databases and similar solutions with high random IO, but for big fat streams a bunch of SATA disks in a big parity RAID still seems like the best bang for the buck.

[Chris Gordon]

Bob (and anyone else)

I’d be curious as to your experience when recovery from a disk failure in parity RAID sets (RAID5, 6 etc). Do you see overall impact to the arrays during the rebuilds? Experience additional failures during the rebuilds?

Thanks,
Chris

[Chris Gordon]

Bob,

I’ll have to respectfully disagree with you here. Some of your challenges with a RAID5 vs a RAID10: (Note when I say RAID10, I mean a stripe across a number of mirrored disk pairs, not the other way around).
– Rebuild times. When you lose a disk in a RAID5, the rebuild on the new disk is often far more painful than a RAID10 considering you have to read ALL of the data on the RAID set and recalculate parity for all of it. With RAID10, you just have to read data from the remaining mirror disk and write it to the new disk.
– If you’re doing a lot of updates to existing files or very small file writes you may not get the advantage of the extra disk spindles when compared to a RAID10 but you pick up the parity calc and parity write.
– Updates of files on RAID5 involve a lot more IO than on a RAID10.

Conversely, there are plenty of cases where a RAID5 or RAID6 is a better solution. At work, I’m doing some disk performance testing right now and moving a 12 disk set from RAID10 to RAID6 gave me a significant performance increase (a couple hundred MB/sec) and more space for large sequential writes. This application is one almost solely writing large files to disk with no updates or noticeable random IO and I’m not too concerned about loss of disks or the array.

Now for the part I think Bob will agree with:

Like everything, which RAID is appropriate depends on the specific situation and balancing a lot of factors. The best way to really know what is best is to test different configurations with your work flow and applications. Of course this costs time/money to do. Video editing is typically characterized (at least the majority of the data) by large files sequentially written to disk (ingesting the video) and then repeated reads of those files or at least parts of them. The additional spindles available for the reads gives RAID5 or RAID6 is big advantage in performance compared to RAID10 for the same number of disks.

Chris