Need a storage/possible RAID solution

[Dave Haynie]

[John Rofrano] “[Dave Haynie] “What they really mean is that the RAID chip, card, or box with its own processor to run the RAID software. Otherwise, that software runs on you PC’s processor.”
Yes, and that’s the point I as trying to make. If your CPU has to run the RAID software it’s going to be painfully slow with RAID 5. “

Yeah… I wanted to point out two things. One, that you really have to check your specs, if you’re buying a device. I suspect just about every JBOD out there has RAID software included, but that’s running on the host CPU, and it’s probably very slow. So when you’re looking at an external box, make sure it’s real RAID with its own CPU, as with yours.

It’s not just the checksum calculation with the external hardware, though. When you do a write to the external RAID, that’s write cached — goes into RAM on the RAID controller board. So the main PC is isolated from the time it takes to complete that write.

There are several things that cause software/firmware RAID to be slow. One is your RAID stripe size. When you write files smaller than the stipe size, the controller has to read-modify-write… it’s reading in the data, making the change, calculating the checksums, and writing it back out. I’ve been playing around with Intel’s latest software/firmware RAID on my new PC, and in one test, I saw writes of 512 byte blocks going at 1.4MB/s, pretty pathetic. And I saw CPU use jump briefly to 12%, which reflects this extra work. About what I saw, at least by degree, the last time I played with this stuff. This is no different from the hardware RAID, really, except that those slow writes are handled by the RAID controller’s CPU… Windows never gets bogged down by them.

This reinforces a hard rule of modern HDDs — large writes go fast, small writes go slow. This pretty much worst-cases it, and thus, if you’re putting in a RAID for something like an internet server — pretty much all small writes, this is lethal.

On the other hand, I looked at larger writes. At 2MB writes, I was seeing 179.8MB/s, which doesn’t suck, and you don’t get much smaller on a media drive. The CPU use didn’t peak above 4% on any of the larger reads or writes, and was usually a 1-2%. I looked that up and found that, while the Intel software is still using the host CPU for checksum calculations, it’s using a hardware coalescer to transform a large number of would-be small checksum writes into a very few single I/O operations. That actually makes sense; the checksum isn’t a great deal of overhead to calculate, but writing it (a small block of data for every large-ish RAID stripe) is a killer. The other thing they did to improve performance is allow write caching (haven’t played with that yet)… applications just write to a RAM cache, and the data gets flushed to HDD later. Very fast, but dangerous if you don’t have a UPS on the system.

Anyway, my main goal was to really find out why these things were so slow… the checksum answer just didn’t do it for me. This was with ATTO… I need a better, more modern disc benchmark, but still kind of a surprise. I still expect to port over the RAID controller from the old PC, once I’m done with it.

-Dave

[Malcolm Matusky]

To RAID or not to RAID, that is the question!

I was using a drive dock with pairs of bare drives (raid 1) that quickly became a PIA. Went to a “hardware” raid mini SAS card and a sans-digital 4 bay enclosure. When I “add or subtract” drives, I do it with another 4 bay enclosure and never touch the drives themselves. I thought about an 8 bay enclosure, but the 4 bay units hold a lot of material and are easy enough to transport or store in a fire proof cabinet, the 8 bay units are too large for that. The enclosures are cheap and I am not overloading my PC’s power supply or chassis cooling system by adding drives internally. The raid 5 configuration works fast enough for me for working with 1080P HD. 4K may be another issue, then it’s time for more high performance controller cards and raid 1 or raid 10 configurations.

Good Luck,

M

Malcolm
http://www.malcolmproductions.com

[Dave Haynie]

You can certainly avoid overheating or underpowering an internal RAID. But what you can’t avoid is that it’s entirely dependent on the PC, on Windows, etc. to operate. That always spooked me — that PC goes down and my files are inaccessible, my really critical ones that I put on RAID to make them more likely to be available. And if the RAID controller is on a main board, do I even know for certain the replacement PCB will have compatible RAID firmware? Nope!

I hadn’t messed with in-box or motherboard RAID for some years, but I’m playing with it now, even though I have the main RAID on a Drobo. Part of this experiment was a need for speed — the Firewire 800 Drobo is too damn slow for editing… ok for long-term storage. And they’re pretty pricey, but have the really cool feature of supporting mixed drive types, and automatically rebuilding on larger drives as necessary.

So I set up a 3-drive RAID5 using Intel’s so-called Rapid Storage Technology, Enterprise edition (well, that’s marketing for you I guess). Worked ok, was slow on small writes as described above, but suitably fast on large writes and pretty much all reads… better than half the speed of the SSD, faster by far than a single 3TB drive.

I was fooling around with boosting the CPU clock (I know, not a good idea, but new system, I’m playing around) and got to a point were I had to reset the BIOS. The system came up in non-RAID mode. Switched to RAID, and it couldn’t boot — something ate the Intel RAID driver, not sure if was a software update or a glitch, but nothing else was missing…. I had the boot drive on that chipset-based SATA, that’s why it wouldn’t boot. Moved that, booted, installed the Intel RAID driver… rebooted. Now the RAID BIOS claimed that 2 out of 3 RAID drives were not RAID drives at all but standard separate drives. Looked up on Intel support sites, pretty much everyone says you’re screwed at that point. So I tried a frightening hack: I deleted the RAID, set up the same structure anew, rebooted. Then I used a partition restoration program, which found the original RAID (which it could, since the controller had the right parameters to run that particular configuration), wrote that, and was back in business. But no thanks to Intel’s software… if you can’t trust your RAID environment, dump it. I’ll be using a different solution on this system for a faster data drive… maybe the Drobo’s not such a bad deal after all 🙂

-Dave

[Dave Haynie]

I’m ending this experiment… in regular use, the occurrence of small writes is just too common to make the “chipset” RAID5 worthwhile, even with the latest extra stuff in Intel’s C600 I/O chip.

My main purpose of RAIDing it was to have a larger, faster “work” drive; I have the slower external RAID for mass storage. My old system has the single 3TB HDD, which had ok performance for this. But with that much storage, I know it’s not getting as much flow-through to the external RAID or backup as it once did, so I still like the idea of RAID for reliability. I’m going to move this to a RAID1 or RAID10 and see if that changes the performance.

-Dave

[John Rofrano]

[Dave Haynie] “I’m going to move this to a RAID1 or RAID10 and see if that changes the performance.”

Let us know how this works out. I was very disappointed with my RAID 5 write performance and would consider moving to RAID 1 or RAID 10 if it’s faster.

~jr

http://www.johnrofrano.com
http://www.vasst.com

[Malcolm Matusky]

Scary thought that my Raid may “disappear” because of driver software. One preventative measure I have been taking is to “clone” my C-drive monthly so if the worst case scenario happens, I have a chance of going back in time to a point where the system was stable and working.

Any comments on this strategy? I am interested in making my system as fault tolerant as I can reasonably afford since I only have one edit system to work with. My thinking was that with a cloned c-drive and an external raid 5 array I would have a greater degree of system resiliency than relying on everthing being in the tower.

One thing I did not consider was the controller card! Would a new mini-sas controller card not work with my existing raid 5 array? I would consider buying a spare, identical card, if this would insure my ability to preserve my raid. Any thoughts on this?

Cheers,

M

Malcolm
http://www.malcolmproductions.com

[Dave Haynie]

I wasn’t shocked so much at the RAID5 slowness, once I figured out why it was so slow (small blocks = small writes) on small files. But I was actually surprised by how many things are doing small sized writes to disc, anyway… my system has so much more memory, but many apps seem to think they’re back in the 1990s, memory-wise, at least for file buffering. That’s really what killed it.

But that’s ok.. the RAID10 is rebuilding. I did a couple of benchmarks, and yeah, the RAID10 is faster than the single drive. Tiny writes are slow even on an SSD, but they scale up pretty fast on the RAID10. Here’s my SSD as a baseline:

And here’s the RAID10 (4 x 3TB HDD):

Not too shabby. I’m seeing 90-100MB/s on the restore, limited by the original 3TB WB drive.

-Dave