[Steve Modica]

I think you are using this direct attach so you are probably doing OK. We focus on shared environments where many threads will hit the raid causing performance and latency drop offs.

You can definitely get better than 700MB/sec through that card. We’re getting those numbers with shared loads in realtime, so I would expect 1.5GB/sec with just raw bandwidth.

The Aja test isn’t really a good measure since it uses a relatively small IO size. FCP uses Asynchronous IO and much larger IOs.

Steve

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

This is doable with 10Gb and the 6Gb SATA stuff. The trick is that all of it is pretty raw so you have to test in advance to make sure the pieces will work together (we’ve been working on that for a while).

You used the term “bandwidth”. I’ll caution that this is a bad starting point. With shared loads, you cannot do “bandwidth/codec rate”. The performance and latency curve of a raid knees off very quickly when you hit it with many streams, even very slow streams. It’s also the case that raids are usually set up and tuned to be in a single stream (or “small” stream count) environment. So the individual IO latency suffers to provide a better overall average performance. (for example, IOs will get reordered or data is read in advance, making the IO take longer)

Whatever you buy, you should setup a realistic benchmark and monitor the IOs so you can demonstrate that they stay under the frame rate window requirement. Once you achieve that, you can sort out the network bandwidth requirements and take it from there.
Steve

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

I’m with Bob on this. I would much rather have had a Mac server (since it natively supports AFP and uses a BSD kernel). Then you could profile the IO using Dtrace and see what’s happening with the Samba or AFP daemon. Are the IOs small or larger? What’s the latency on each of them? Depending on the drives, I’ll wager you see “lumpy” numbers that are very good most of the time, and very bad some of the time.
Most vendors focus on drag race benchmarks (which is what you posted as well) and those are not relevant when dealing with a shared realtime load.
I’m curious as to what you’re TCP statistics look like. I’ll wager you are seeing a lot of SACK events on one side or the other. That will be another problem and probably explains your 1Gb performance numbers.

Steve

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

The raid vendors MB/sec numbers are not going to be the same when you start a shared load. You’ll find that it performs much slower since the IO is not sequential. The tuning is also different since things like read ahead can hurt rather than help in shared environments.

Given the raid you have, I don’t think you need more than 2 Gigabit ports because they will top out that raid. You could go to a 4 port card but to get the stream counts we get, you’ll need something with more spindles and a higher stream count capability in a shared setup.

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

The R680 card can work with 3Gb drives and offer better performance but you have to get the firwmare right and the expander has to be correct (yours won’t be).
The seagate 1.5 TB drives wern’t good realtime performers when I tried them. (I’m not sure if you are using ES drives or not, but even their ES drives didn’t do well with shared loads.

The 680 cards will work well with the right pieces, but probably not with legacy pieces.

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

SNS is big into iSCSI and I know they know those target stacks well (as well as their iSCSI initiator).

The thing about any of those solutions is that you get a linux black box. When you want to upgrade, you need to throw that away. I like to have people use a Mac as the server. When they upgrade, it moves to someone’s desk. It’s very economical. Less landfill etc.

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

6gb SATA supports a multiplexing option that will let us handle 2 3Gb drives on a single 6Gb channel. That’s important to getting the benefits. If you have a 3Gb expander now, you won’t get that.
We’ve also found that the 6Gb Atto card (which is a completely different chipset from the 380 card) requires different numbers to perform the way we want.

I would not suggest upgrading to that card. I think Bob is right that a 380 would be sufficient for what you have as a “better” replacement controller.

We have 6Gb stuff that does all the right multiplexing if you want to find out more about that.
Steve

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

Of course you can do this. We do it all the time with our 6 port cards!

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

Sorry to be late to the party on this.
Some comments:
Link aggregation is good for many to one sharing scenarios (eg 6 ports, 20 clients). If you have 4 ports and 5 clients, just buy a 6 port card and give them each their own port (do not use the apple built ins because they have a nasty habit of queue stalling right now. I’d even put our cards in the clients.) The secret is that we use older chips and the errata list is very well understood. They are line rate and extremely robust.

The switch can be skipped and you can go direct. If you wanted to use a switch, Netgear is not the right choice. Edge-core is the least expensive thing you’ll find that will do flow control correctly for this application. That’s why we sell it. We offer the very lowest cost thing we can find that does it right. (And we’ve been through many firmware iterations with them BTW). Most vendors default flow control off and so they don’t see many bugs on that feature. Thus it’s often broken.

The Sonnet chassis looks like a Small Tree chassis. That’s because the same company bends the metal. That’s where the similarity is going to end. Almost no drives on the market will deal with a shared realtime load. Your chassis will have the wrong drives.
Most of these products are setup for drag races (one very fast stream). Shared storage is very different.

Steve

Steve Modica
CTO, Small Tree Communications

[Steve Modica]

I take this to mean that Xsan is next on the chopping block. Obviously that’s just my opinion, but if you *must* have redundant Metadata controllers and you can no longer buy a 19″ rack mount solution, the enterprise isn’t going to buy it. Mac Pros don’t have redundant power either. It seems like a shift away from that space.

In my opinion, NAS has always been cheaper, easier to scale and safer. Who wants to send sensitive inode operations out over a cheap 3COM switch using TCP? (Oh yeah… Xsan) 🙂

Steve

Steve Modica
CTO, Small Tree Communications