Chris Murphy
Forum Replies Created
-
I’m not recommending raid5, maybe raid6 is a good fit. But you asked an open ended question without any detail on the use case, work load, backup strategy, or the drives being used. So I’m just poking holes. Maybe raid6 fits your use case exactly right.
RAID6+hotspare is a red flag to me, that says, “this data is really super important and needs to always be available”. And it necessarily implies the workload can still get by on degraded performance, or you can tolerate waiting for the rebuild. Video production workloads are demanding as is parity array rebuild, so I’m skeptical that you can do both in a 1x fail raid6. But I have to defer to more experienced people on this question. For a 2x fail raid6, the performance must obviate doing any meaningful work.
Let’s look at the rebuild times. The Promise web site says “Pegasus2 R8 32TB model is populated with 5900 RPM SATA drives” which is why I asked what drives you’re using. Those are probably Seagate Barracuda XTs, which average about 130MB/s on sequential writes meaning it will take 30 hours to fully write. Raid1/10 can rebuild at the drive’s max sustained write speed. Parity raid will take longer. How much longer depends on the controller.
So there are three questions: Is the 1x degraded raid6 performance still decent enough to get work done? If not, can you tolerate the downtime for the rebuild? And how much longer than 30 hours is it?
There are all sorts of ways to resolve this. The extremes are getting 10K-15K SAS drives of smaller capacities so the rebuild times are shorter and the performance is better, except the Promise spec sheet says drive support is SATA, not SAS, so that’s maybe a dead end. The other is raid10 which gets you a much smaller performance hit in degraded mode, with even a 3x drive failure, let alone the 1x failure you’re most likely to encounter, the rebuild time is also faster.
Also, raid6+hot spare means you’re setting aside 37.5% of your storage capacity. And another 12.5% hit isn’t much compared to the gain of raid10 over raid6+hot spare.
With regard to raid10 vs 0+1: Use raid10 over 0+1 because it always rebuilds faster, and you can lose more drives than 0+1. Their performance in normal mode is the same.
You can relax your concern whether drives are better off spinning or on a shelf. Consider the manufacturer themselves will have a pile of a given drive model in reserve for years. I don’t consider raid5+hot spare to be raid6. And I’m unaware of an implementation that permits a hot spare to be a working rw mount that is suddenly yanked from the user without warning, destroyed, and used in a rebuild upon a single disk failure calling it to duty.
-
I’d think TSO overhead would be less than the layers that are normally done on hardware that have to be done in software with this implementation. The lower block size translating into smaller packet sizes, even if they’re close to the same size, means more packet metadata needs to be generated, transmitted, and parsed. Also, between network and application layer, the page size is 4KB on x86 so pushing around 1K block sizes may actually produce worse results as moving data at less than page size is probably not so well optimized. But I still think it’s a valid test to see if it has any effect at all.
-
Umm, pretty sure MTU still applies no matter what. This isn’t a case where we can skip physical and link layers by bonding two network layers together and just share large blocks. That the physical layer is virtualized doesn’t mean we can break the laws of ethernet and share 64KB packet sizes at that layer. I don’t think.
-
Does this virtual NIC support jumbo frames? Maybe I missed that part.
We know that Thunderbolt 2 can produce a sustained high transfer rate greater than 1000MB/s. And we know a hardware 10gigE NIC can produce a sustained high transfer rate also. So the problem seems to be somewhere in the virtual NIC implementation which is in the kernel.
Linux has been doing TSO in paravirtualized NICs for a while, and getting well more bandwidth between virtual machines than is needed for the use case in this thread. So it seems it could work here too. But it’s also a matter of optimization.
-
TSO is typically done in hardware, hence “offloading”, meaning the CPU doesn’t do it. In this Thunderbolt IP bridge arrangement, there’s no hardware to do the TSO in this case, it’s done by the CPU no matter what. So it’d only be a choice of where the segmentation is being done, and it stands to reason they may have more efficient segmentation code at a lower level, so they simply don’t allow it being disabled.
-
So yes you’re right raid5 and raid6 and raid0 have no read penalty, but to compute the performance factor you only count data drives. A five drive raid0 has five data drives so that’s 5x reads. A six drive raid5 also has five data drives (yes it’s distributed parity but for any full stripe read one of the drives produces a parity chunk which is not read during normal operation). And a 7 drive raid6 likewise has five data drives. So for reads, those raids should perform the same all other things being equal. They each have equal number of data drives, but unequal total drives.
-
Yes, because eight drives in raid6 with a hot spare means stripe members (the ones we get data chunks from during full stripe reads) is five. Parity chunks do not count, and that’s 2 “drives” worth. And the 3rd is the unused hot spare.
-
Maybe one of the network gurus knows whether Apple’s gigabit ethernet hardware uses TCP offload. In any case the physical and link layers have been dealt with in hardware for some time. In data centers much of TCP/IP is also crunched, or at least “pre-digested” by the ethernet card. Otherwise the (general purpose) CPU has to do a whole lot more work.
It’s a totally different implementation, but you can kinda see an ethernet card as the network equivalent of what a GPU does with graphics. If we didn’t have a GPU the CPU would have to do all of the graphics rendering and it would be very very costly if even today’s CPUs could keep up (they can’t). It’s not comparable in that the whole network stack in software with a general purpose CPU is doable, but my expectation is that this is just super expensive CPU wise, and why it’s erratic is because it’s subject to being pre-empted by the kernel for other time sensitive tasks vying for CPU time.
-
re: 2) 6 core nMP in DAS mode attached to Promise P2R8 RAID, 1077MB/s writes vs 817MB/s reads. I see the scren shots, but it seems like reversed numbers. Anyone have an explanation?
As for why IP over Thunderbolt is erratic, that’s a matter of how Apple’s implementing it. If they’re emulating ethernet or something like InfiniBand/RDMA in software it’s probably a huge CPU and memory hog. It’s a PCIe bus, so imagine taking a plain PCIe cable from computer to computer. First I’d kinda expect that to fry one or both logic boards, but aside from that, there’s no mechanism for them to communicate anything this way so that has to be coded somehow. That comes well before SMB, and if it is SMB being used, then it sounds like they’re emulating ethernet in software. Very expensive to do that. It’s not like these ethernet cards have junk on them, or generic purpose chips. They’re specialized and the 10gigE ones have heat sinks on some of those chips.
Can you repeat this bridge test and take a screen shot of Activity Monitor set to All Processes with the %CPU column clicked on? Or even better would be to open Terminal and use:
top -s10 -ocpu
Wait at least 10 seconds for it to update, the initial display is not sorted. Then take a screen shot. BTW I see the screen shots above but when I click on them I get a 404 error, so I can’t see them bigger than they are inline. Any ideas?
-
What’s this for? Video production work or as a backup? What drives?
The raid6 RMW penalty is significantly worse than raid5, plus the hot spare means you’re at best 5x read speeds, which depending on the drives will vary alot unless they are short stroked. Once even one disk has failed, the degraded mode performance I’m very skeptical will supply what you need to actively use the array while the rebuild is occurring. ISo I’m not sure what the extra drive redundancy gets you. In any case there should be sufficient regular backups that you shouldn’t need to depend on raid6. If you’re going to forego regular backups and think raid6 bridge that gap, I think that’s a mistake.
As for hot spare, forget it. It costs you both performance and capacity and isn’t worth it. Before raid6 I’d consider raid10. Before raid6+hotspare I’d consider raid15. But before all of that, make sure you’re doing regular scrubs on the array, use drives that have proper error timeouts so their errors are actually corrected during scrubs, have one or two same model drives on hand and properly replace them when the time comes. If you don’t do those things you can still get bit in the ass with raid6 and a hot spare. And if you don’t need the extra capacity this layout implies, get an R6.