Too early for copper 10GbE?

[Bob Zelin]

Tasked with upgrading the ethernet NAS shared storage for our tiny 3-bay FCP/mgfx house, I’ve been boning up and want to confirm one of my conclusions with anyone willing to comment here:

REPLY – as always, you want to “build it yourself” without involving a contractor, or dealer, and want to do this “as cheaply as possible”. If you keep thinking this way, you will fail. Maxx Digital, Small Tree, and EditShare ALL have working 10Gig ethernet solutions. If you want COPPER over the entire network, you will need one of the newer RJ45 10Gig switches, and the 20 port switches cost between 15 and 20 grand PER SWITCH, not including the 10Gig NIC cards in your MAC computers.
Keep reading …………

I’d originally hoped 10GbE over cat6 would be far enough along by the time we needed up upgrade, but:

REPLY – it’s ready right now.

1. Small Tree copper 10GbE cards @ 1099 each are minimal cost savings compared to fiber (HBAs similar, but installation and software requirements exist for fiber, regardless of protocol)

REPLY – yes, and this card requires a 10 Gig SWITCH that uses RJ45 ports, and these are expensive (15 – 20 grand).

2. The only rj45 10GbE switch I can find (this one) at nearly 20 grand is close to the entire cost of a reasonable fiber system, installed

REPLY – a fibre system is more than 20 grand, and yes, this is the correct price. A 4 client 10 Gig system using copper only, that is expandable to 10 clients (just add more 10Gig NIC cards to the next 6 computers) is $40,000. I can do “combo” systems that use just a couple of 10 Gig ports, and the rest regular ethernet for less money, but it’s still $25,000 – $30,000 with the server computer. Regular gig e systems are much cheaper to build.

3. I cannot seem to find any multi-port 10GbE HBA/NIC cards that are copper, and there aren’t enough slots in a mac pro (all OSX here) for single cards

REPLY – that is correct – you use a multi port SFP+ 10 Gig card, and go into the SFP+ ports in your $15,000 10Gig switch, link agg, and then use the RJ45 10 gig ports to feed your clients. all of this adds up to a LOT of money. There ain’t no free lunch.

4. cat6 adapted from XFP (like for the Small Tree switch) seems kludgy and still cost-prohibitive

REPLY – how much do you want to spend ? You can’t build an 8TB 1 Gig shared storage system for less than $11,000 – a 16 TB 1 Gig ethernet is about 16 grand. Fibre costs more – how much do you think 10 Gig is going to cost you – 10 grand ? WHAT IS YOUR BUDGET ??

The most important thing – THE MOST IMPORTANT THING – is that YOU are not going to be the “hero”. You will hire someone – be it me, Maxx Digital, EditShare, Apace, Cal Digit, Small Tree, a good contractor, or SOMEONE to do this for you. You will NOT figure this out by yourself, and you will not spend 10 grand to make it happen – it will cost more money. 10 Gig over copper is NEW TECHNOLOGY.
10 Gig ethernet has up to this moment only been done with Twinax, CX4, and Fibre cable, never realisitically with CAT6A RJ45 cabling.
Now that this is finally possible, and available, it’s EXPENSIVE. When will it be cheap – not any time soon. And the minute it becomes “cheap”, it will be integrated into the motherboards of MAC’s and PC’s.

(I should mention that at least one > 50ft run leads me to believe CX4 is not an option)

REPLY – that is correct, both CX4 and Twinax can only be run a short distance. You want 50′, you need CAT 6 or Fibre (Fibre over 10Gig ethernet).

Given that… I can’t seem to justify 10GbE over (surprisingly) cheaper fiber systems, despite it being arguably faster, more straightforward to install/maintain, and I suspect MUCH more commonplace in a few years.

REPLY – How much is a fibre system ? I know – do you ? I can build a non expandable 10 Gig system with fibre cable for 27 grand. And if you want an expandable system with copper RJ45, it’s closer to 40 grand. And if you only want one 10 gig client, and the rest regular ethernet, then I can do it for 24 grand.

A $20,000 fibre system is not $20,000. None of these systems based on ATTO FastStream or CalDigit Super share ever mention that you need SAN MANAGEMENT software from Tiger Technology, XSAN, or Command Soft Fibre Jet, which cost $995 PER CLIENT, so it adds up very quickly.

Am I missing anything? I want to believe 10GbE is the next gigabit ethernet, but right now today, it feels like it’s either fiber or f*ing nuts

REPLY – you are not f#$%ing nuts – you are just innocent. You need help. Hire help, hire a contractor that knows what he is talking about.

Bob Zelin

[Shane Winter]

Wow, thanks for the lightning fast response….and first let me thank you for putting so much info out here. I’ve reaped the benefit from volumes of your guidance and discussion here. So…thank you!

Second let me clear up that I in NO WAY intend to do this myself…we have and use great vendors. Nonetheless, I still want to understand the options as comprehensively as possible, to make the best decision for my company. So I hope some general questions are appropriate for the forum here, and especially that it doesn’t sound like a request for free consulting: trying to keep it general.

I guess what I’m after is a gauge of whether 10GbE via cat6 is worth it’s extra cost….

“I can build a non expandable 10 Gig system with fibre cable for 27 grand. And if you want an expandable system with copper RJ45, it’s closer to 40 grand.”

Very much like my numbers. So…extra for copper, yes? Are you including software costs here?

“None of these systems based on ATTO FastStream or CalDigit Super share ever mention that you need SAN MANAGEMENT software from Tiger Technology, XSAN, or Command Soft Fibre Jet, which cost $995 PER CLIENT, so it adds up very quickly.”

I’m well aware of this and it’s what makes me hesitant to bite on fiber in the first place…both for the immediate cost and because we don’t (yet) employ full time IT staff here. The simpler the system is to manage, the better.

“you are just innocent. You need help.”

Boy are you right on the money there.

It sounds a bit like the gist of your answer is that it’s doable but expensive…is that a fair paraphrase? If more expensive, my partners and bottom line demand an answer to “why?” and this is where I’m having trouble.

My gut tells me that speed, simpler infrastructure, and imminent popularity make cat6 a good idea, but I haven’t worked with it (at 10Gb anyway), and worry that I’m barking up the wrong tree. If I can’t show a real benefit, then I’m obligated to go with the more affordable fiber systems quoted to me.

[Chris Gordon]

I second all of what Bob said. Definitely get some help from someone that specializes in this — they should know what really works and solves your problems and what is vapor-ware. Another thing is to work try go with a vendor that can sell you everything you need (array, switch, NICs/HBAs, etc) so you can get some better leverage of a volume purchase. If you call up and order just a single switch, you’re going to pretty much pay list price. If you are bundling a whole bunch of stuff together, you have a lot more negotiating power to get discounts. Of course, always shop around and get some comparative prices. Don’t be arrogant or mean with your vendors but nothing wrong with getting designs and quotes from say two different ones and being open with them that you are also considering their competitor. Again, a good consultant should be able to help you with this.

As for 10GigE, it’s expensive. I’m actually working on a project at work with 10 GigE (not video related) and it’s just not cheap.

[Shane Winter]

original reply showed up 5 hours later…so I’ll remove this retyped one. 1000 apologies

[Shane Winter]

Duly noted…and again, I will be paying through the nose for very competent people to do this for me. Fear not.

So…expensive, yes, but evidently worth it in your case. What determined this for you? Raw bandwidth requirements?

[Chris Gordon]

IMHO, 10 GigE over copper is still just way too new. You pay a serious premium for anything really new and there are bound to be some “adjustments” to the “standards” as the new tech settles in. I’d at least expect to run into some odd bugs and have to do a number of firmware updates to the switch and NICs. Unless there is some real need to use the very newest tech, 10 GigE over copper in this case, stick with something more tried and true. Let others pay the early adopter tax and figure out all of the bugs for you.

[Lorenz Redlefsen]

(Full disclosure: I work for an Ethernet switch vendor, in the engineering department.)

2. The only rj45 10GbE switch I can find (this one) at nearly 20 grand is close to the entire cost of a reasonable fiber system, installed

There are plenty of “rj45 10GbE” switches shipping in volume today. The keyword to search for is “10GBASE-T” — that’s the standard that defines the PHY layer for 10GbE-over-twisted-pair.

Note that most (but not all — check your vendor’s datasheet carefully!!) 10GBASE-T switches allow their ports to run at multiple speeds, using autonegotiation to figure out what speed to run at, i.e., 10G / 1G (and some even support 10G / 1G / 100Mbps — again, double-check!). If you’re interested in consolidating your network infrastructure and connecting hosts running at different speeds to the same switch, 10GBASE-T might be a good option.

On the other hand, there are non-10GBASE-T copper options out there that might be less expensive than fiber, at lengths of up to about 7m. These are mostly interesting for connecting rack-mounted servers to nearby (e.g., top-of-rack) switches. They are pre-terminated “TwinAx” cables that use SFP+ connectors at both ends.

There’s no IEEE standard for this media type — my employer calls it “10GBASE-CR”.

You’ll need a switch that supports SFP+ ports and 10GBASE-CR in order for this to be an option. The nice thing about a switch with SFP+ ports is that you can mix-and-match media types (long-reach fiber, short-reach fiber, passive copper, …).

Some (but not all — again, check your vendor’s datasheet!) SFP+ switches support 10G and/or 1G SFP modules, so here, too, you could consolidate your older 1G equipment into the same switch.

[Shane Winter]

ahh…the 10GBASE-T clarification helps immensely

I had definitely considered the SFP+ route, mostly because I couldn’t find anything else. kept coming back to high cost numbers by the time I added in connector/tranceivers and cabling, though (again, this in comparison to relatively few fiber runs)

Consolidation is one of the reasons I thought it might be worth it

[Steve Modica]

10Gb ethernet with 10GbaseT phys is still expensive. The PHYs (physical layer chips) are discrete (separate) chips and they use a lot of power. So switches can’t be too dense or they will melt. Cards require extra space and chips so they use a lot of power, have higher failure rates and are expensive. However if you need it, it exists.
Shortly (as in, we have samples and the driver work is done) there will be integrated PHY chips. The MAC (Media Access Controller) and the PHY are one chip. They use very little power and will allow the prices on both cards and switches to fall.

Personally, I think gigabit is still a great value and if you run cat6A everywhere, you will have no trouble upgrading. Since gigabit is so cheap, what’s the risk? You might toss $2000-3000 worth of cards and switches when you go out and buy your $10000 10Gb switch? You can always repurpose that stuff for print servers or something 🙂 Additionally, RED keeps rumbling about these super low bit rate codecs. If all that stuff works out, Gigabit has a lot of legs.

In Q1 or Q2 of next year, all the 10GbaseT stuff will be hitting.

One caveat. FCoE (we have the driver going up as a free beta on our website soon) requires a certain bit error rate and 10GbaseT does not meet that spec. So it’s likely FCoE will work and be supported (we support it now), but there will be some length limitation to help deal with that bit error rate requirement. FCoE and other storage protocols are very sensitive to errors. This is one thing TCP gets right. It expects errors. FC and SCSI do not.

Steve

[Lorenz Redlefsen]

Steve,

I think at the high level, we’re in agreement: 10GBASE-T is ready for mass-deployment today, albeit at a price premium to 1G. When you do an apples-to-apples comparison, I don’t think that price premium is as bad as you make it sound, though.

(I don’t think it’s fair to compare the per-port cost of an el cheapo 10/100/1000 switch with an enterprise-class 10G-capable switch, with redundant power, cooling, etc. It *is* true that there aren’t any el cheapo 10G-capable switches today, so 10G does not compare favorably when you compare “cheapest-to-cheapest”.)

10GbaseT […] PHYs (physical layer chips) […] use a lot of power. So switches can’t be too dense or they will melt.

My employer has been shipping a 48-port 10GBASE-T switch with discrete PHY chips for over a year, so I’m not sure that this is entirely accurate. Yes, it was a challenge to fit all those components into the system — maybe that’s why we’re the only one who has managed to pull it off. 😉

The density limit almost everyone eventually runs into is that you can typically fit only 48 RJ-45 jacks into the front-panel of a 1RU rack-mountable switch. It’s just that we ran into that limit before our competitors… 😉

Shortly […] there will be integrated […] MAC (Media Access Controller) and […] PHY [solutions].

That’s true for NICs, but remember that on switches, the MAC is usually part of the switch chip, while the PHY is an external chip. If history is any guide, that’s likely to remain true for the foreseeable future. Note, for example, that even today’s 1000BASE-T switches still use discrete PHY chips, even though they are multi-port — you usually get 8 10/100/1000 ports per PHY chip these days.

One caveat. FCoE […] requires a certain bit error rate and 10GbaseT does not meet that spec.

My understanding is that both Fibre Channel (FC) and 10GBASE-T require a BER of 10^-12 — which BER spec does 10GBASE-T not meet?