10gb Experiences?

[Sean Oneil]

Shane,

That’s correct, we were not talking about a SAN. If I were going to do a SAN system today I personally would choose fibre channel rather than 10gbe with iSCSI or AoE. I agree with you there. But we were discussing AFP file-sharing, which is not possible over Fibre Channel. There is a protocol for “IP over FC” as you mentioned but it doesn’t exist in the Mac OSX kernel and surely never will. If it did exist, I’d be using it right now.

Yes AFP is less efficient than block-level SAN. But it’s also a lot less complicated, less expensive, and makes it much easier to use your own storage devices that can easily be upgraded down the road. It might make more sense for smaller places.

The reason Bob was only getting 180MB/s is surely the limitations of AFP on OSX Leopard. If one were to use NFS sharing over 10gbe on Linux or Solaris, it would no doubt be 3 times faster at the very least. Based on what I’ve read, AFP is inefficient and the TCP stack in Leopard sucks. But I’ve also heard that Snow Leopard is going to drastically improve this. We shall see. I think people at large facilities will still be using SANs for years to come. But when 10gbe becomes more common I think it’ll be more than enough to satisfy the needs of small shops.

As far as copper vs. fiber interconnects, it’s just a matter of distance. All the Macs at my place are next to each other in a machine room, so if I were to get 10gbe adapters I’d only need the copper CX4 ones.

Sean

[Bob Zelin]

Shane –
do you know why people discuss this so much. Because people don’t want to spend any money. People want cheap. And they want easy. Do you know why they want easy ? Becuase they don’t want to hire people like us. They want a cheap system, that will run without administration. This is why XSAN is a failure (relatively speaking). No small company wants to deal with it, unless they employ you or Mark Raudonis.

CX4 can only run 15 meters. CAT5a promises a normal 100 meter run for 10Gig ethernet. I have NEVER EVER run cables (including audio and video) more than 200′ at ANY facility I ever worked at in NY or Florida, so the idea of fragile, expensive Fibre (along with all the stuff that goes along with it) means nothing to me. However, I like staying employed, so COST is the #1 factor – performance (like enough to handle DPX files) is very unimportant.

If you observe the AV industry, CAT5e and CAT6 has taken off because they don’t need expensive double shielded coax cable for hi def, and they can use “telco installers” to do the wiring, instead of video professionals. Labor is very very very important in decision making for corporations, and the ability for “Joe Schmo” to install Belden 10GX for 10G transmission is A LOT MORE APPEALING than the trained expensive guy that knows how to terminate a fibre cable. 10 Gig will win – end of story. I know fibre is teriffic.

But that’s just my world.

Bob Zelin

[Shane Sokolosky]

I still think it’s less expensive and less complicated to go with fibre, although your initial investment was not fiber, but you could get a 4Gb fibre RAID for the same amount you paid for the Cal Digit.

The only components that you’d need after that to have a 3 seat SAN would be :

3 Seats of MetaSAN
3 Apple HBA’s (copper cables included)
and a Qlogic SANbox 1400 switch

and only costs

about $6,800

From a tech perspective 10Gb E is cool and high tech, but from Bob’s perspective I don’t see ethernet being any easier. this is a REAL solution for you if you REALLY wanted to do what you originally were trying for with 3 people editing HD. I also bet if you wrote an instruction manual on how to install this stuff it’s be shorter than Bob’s tutorial here on the cow.

If we don’t want to put 7k towards a solution to take full advantage of a HDCAM SR Deck and 3 editing stations then we might want to rethink a strategy altogether.

Shane Sokolosky

Consultant / Systems Engineer

XSAN for Video Apple Certified Technician
Apple Consultants Network – Storage Area Networks
Apple Developer Connection

SANtech.TV
Office: 714-639-3767
Mobile: 714-599-1611

shanesky@santech.tv
https://www.SANtech.tv

[Sean Oneil]

[Shane Sokolosky] “4Gb fibre RAID for the same amount you paid for the Cal Digit”

The CalDigit RAID card costs $470. I would provide storage I already own.

[Shane Sokolosky] “3 Seats of MetaSAN
3 Apple HBA’s (copper cables included)
and a Qlogic SANbox 1400 switch

and only costs

about $6,800”

Plus the FC storage, which would be over $15k for the speed we’re talking about. So like $22k total

[Shane Sokolosky] “From a tech perspective 10Gb E is cool and high tech”

That’s pretty much all I intended this discussion to be about.

[Shane Sokolosky] “Bob’s perspective I don’t see ethernet being any easier.”

I’m totally baffled by how you could possible think that!?! I used MetaSAN for two years. I still have four licenses of version 2. It’s vastly more complicated than sharing storage over AFP. And you need a dedicated extra network for metadata. And then of course there’s configuring the FC switch. I’m not knocking it or calling it overly difficult. But to say it’s EASIER than AFP file sharing – that’s crazy.

[Shane Sokolosky] “this is a REAL solution for you if you REALLY wanted to do what you originally were trying for with 3 people editing HD”

I totally agree. But I’m not shopping to address an immediate need. Sorry if I wasn’t clear. 10gbe adapters suddenly became within the price range to where I would buy a setup like that now just for the luxury of it – if it worked well. It doesn’t, so I’m not.

Sean