-
Why 10Gig Ethernet (copper) is “worth it”
When I first looked into 10Gig ethernet, I was surprised what I saw – I foolishly expected the “next gen” of RJ45 CAT6A products – well, this is not what 10Gig was about (RJ45 CAT 6A is just getting started – FINALLY !). 10 Gig was transmitted in short distances via Twinax copper (max distance 10 meters), CX4 copper (max distance 15 meters) and Fibre. Fibre ? What the hell is fibre doing here – I want 10 gig ethernet ! Fibre up until recently was the only way to do long runs of 10 Gig ethernet, which meant that you had to have fibre cable AND fibre transceivers (and not cheap ones – SFP+) on both your client card AND on your switch, or multi port 10Gig card. And BOY, was all of this expensive.
There have been for a while “tricks” where you can get a Gig E switch (level 3) with a couple of 10 Gig ports in them (the rest regular Gig E) and have one 10 Gig client with the rest being regular Gig E (EditShare does this with the HP Procurve 2910al switch). And it works. But you want MULTIPLE 10 Gig clients, AND you want copper because you want backwards compatibility to your normal computers.
Well, yes, it’s expensive. RJ45 10 Gig switches are brand new, and expensive – the cheapest one I am aware of is 15 grand. But you get to use CAT 6 cable (for up to 55 meters) and you can plug in your regular MAC computers with regular Gig E and it will still work. A RJ45 10 Gig card ranges between $500 and $1200 depending on your vendor.
So, why do this – NO MANANGEMENT SOFTWARE – you get to use Apple File Sharing. It’s easy.
If RJ45 10 Gig switches cost only $2000, EVERYONE would be doing this – and when this day happens, Fibre will disappear quickly, and now protocols like FCOE will take over as well.
Bob Zelin
-
11 Replies
-
The other interesting development is Converged Networking for which 10 GigE over UTP is a strong enabler. With converged networking, you essentially virtualize all of your connections (ethernet, FC, etc) over a single large (i.e., 10 GigE) connection, or 2 connections if you want redundancy. Cisco is already pushing this in their UCS blade servers and you can get CNAs (converged network adapters) from a number of different vendors. You can do some of this today without CNAs, but not as cleanly. I think this is clearly the direction we will all end up going, but its still a couple of years out, especially for affordable CNAs and switches.
Bob, have you looked at any of this and have any opinions/thoughts?
-
Worth nothing that fiber-based 10GbE is similarly appealing: Looking like we’re gonna be able to use Small-Tree’s 4 port card to serve up direct 10Gb ethernet connections to a small number of workstations without needing a switch or SAN software. With the ability to get pre-built fiber cables easily and cheaply, installation isn’t much different from cat6 or other copper, and range is excellent
-
I have done the math on this, and with a 16 Terabyte drive array, and MAC computer to act as the server, it costs $26,767 to do this (not including installation), and remember – you have a 4 port card with NO switch, so there is no expansion beyond 4 clients with this configuration. Now, 27 grand is still a pretty competitive price, but the equivalent GigE system is 10 thousand less than this – and most people almost faint if they have to spend even that.
You know what the big problem is with all of this stuff – THE PRICE.
In the same way HD VTR’s were the big “problem” a few years ago, todays big problem is hi speed data transmission. Money, not technology.Bob Zelin
-
I’ve chatted with Bob on this forum about this before and I think we are very close. I’m wiring my new office up with CAT6 in anticipation of 10GB over copper RJ45. The ‘holy grail’ for me is an affordable 10GB switch in combination with a fast affordable array (I don’t care if it’s miniSAS or 6GB Sata… as long as it’s fast and cheap). I want to serve both edit and graphic stations as equal citizens and not have to constantly copy graphics and edits between local RAIDs.
I’d easily pay $5K for a 10GB switch – those things will last 5 years easy, at $1K per year of investment that’s a no brainer. This is the company I’ve been keeping an eye on:
https://www.aristanetworks.com/en/products/7100t
Founded by Andy Bechtolsheim – he’s pretty much a legend here in SIlicon Valley, co-founded Sun and Arista seems to be trying to bring affordable RJ-45 hi-density 10GB to the masses. These switches have Sun written all over them (modular fans and redundant power supplies) – even the hexagonal airflow vents are Sun styling. They will look nice next to my Xserves (don’t get me started on this…).
Anyway, not sure what the street price for the Arista 7100T, I saw a forum post ~$13.5K… not sure if that’s accurate. Really anything a click or two under $10K would make me think very seriously about pulling the trigger.
I seem to remember that some of the issues with RJ45/copper had to do with power and heat – hopefully these have been solved.
-Greg
Greg Leuenberger
CEO
Sabertooth Productions, Inc.
http://www.sabpro.com -
Hi Greg,
that is very astute of you to find this product. Yes, this is the holy grail switch right now, but it’s 15 grand, which is more than most are willing to pay. I can put together a 4 seat 10 Gig system using this switch for 40 grand (and others can plug in their legacy 1 Gig stuff too). The single port 10Gig RJ45 card is a little less than $1200 each, so this stuff gets expensive. The rest is the usual MAC server, 16TB drive array and ATTO card.Yes, if this switch was only 5 – 6 grand, then bye bye fibre channel.
Hey, give it 3 years. I am sure we will see the “usual suspects” charging this. The Arista supports link agg, so we can pull of the same tricks that we currently do with ethernet.
Good job, Greg !
Bob Zelin
-
It looks like a contender, but you’re right – $15K is too much right now. Hopefully we won’t have to wait 3 years – in 3 years LightPeak may be out and an option (I think it can handle long cable runs). I suppose it’s too much to expect it to drop much in 2011 though (fingers crossed – I’d love to see a 12 port switch, enough for 8-10 machines…edit, render and graphics). I wouldn’t rule it out, 8-12 port fiber switches are common.
-Greg
Greg Leuenberger
CEO
Sabertooth Productions, Inc.
http://www.sabpro.com -
I can’t wait for this stuff to come down in price. I have been running 10g for a year now. It’s currently just two stations sharing peer to peer at 300mb a sec. I have this setup becuase with smoke on mac and resolve the hardware is different and has to been seperate systems. It has changed my workflow so much. I’m never waiting for copies. Both systems have local raids and I can mount them on any machine over afp.
I have been thinking about adding a 3rd machine with a 6 port card. Is it possible to link two ports and get more bandwidth? Trying to figure out if I could do uncompressed 3d hd over ten gig. This would be to only one user.
Nick Hasson
Smoke/RESOLVE
http://www.niceedits.com -
YES ! Let’s do it Nick –
but I am billing you this time !Bob Zelin
-
However, none of the converged implementations are what I would like to call terribly stable nor would they be considered to have a full backplane. I’ve seen and demoed all of their gear at this point as well as full customer testimonials on how much they hate it.
Some of the stuff coming down the pipe from some vendors will let you retrofit a rack and plug in their provisioning software. My money still tends to stay on stability over features.
-
You can do a raw fcp path with no management software. However, typically this management software is intended to dealing with multiple paths for redundancy. You can’t do that with with Ethernet without an expensive set of switches which have the fail over capability. Which is not something you are going to see on a low end switch.
The other issue with traditional ethernet is the performance gaps from the stated maximums. Not only do the more affordable cards rarely actually achieve their theoretical maximum, but when dealing with the layer 3 side there is protocol overhead.
Ethernet is ok and the protocols which stand on top to emulate storage are not bad either. However, there is always going to be overhead which makes it less then ideal. It’s not so much the copper, but the philosophical differences between their implementation.
When dealing with bonded links it’s important to remember that it’s not truly a single pipe. An LACP link is going to have a packet transmission path based on whatever sort mechanism you choose. (Typically source/destination hash). This means that while the bonded link might appear larger there is going to be a throughput limit between the same source and destination equal to the smallest link in the aggregate. It may or may not be a problem depending on your use case, but it’s important to be aware of how that bond really works.
So 10GB (which I can do cheaply on fiber today) can do great things with disk networks. However, it’s not going to be as versatile as 10GB E on copper tomorrow. (And really you only do fiber ethernet if you have distance issues or problems with current sink due to negative ground potential difference)
Reply to this Discussion! Login or Sign Up