What happened to Infiniband as a SAN topology? Especially on OS-X?

[Bob Zelin]

I know that I am going to regret responding to your post.
Here it goes –

I’m a DIY kind of guy (yes I know, yelling in 3.2.1) and want to learn more about shared filesystems and SAN’s. For reasons I can’t understand, I have an irrational dislike for fibre-channel, not a serious one that I can’t get over but nevertheless from my linux days, I’m really curious to setup a little IB 3 node test SAN now that the IB players are all going to 20Gb/s and 40Gb/s and tossing their 10Gb/s stuff onto Ebay for pennies.

REPLY – what drugs are you on pal ? 10Gig is FINALLY becomming a little mainstream. Tossing it on ebay ? It’s seeing light of day for the first time in mainstream, and it has not even hit the motherboards yet. 20Gb/sec – from exactly what vendors – show me a link that I can BUY these products. And why do you dislike fibre channel – because it is too expensive ?

So before you yell at me for not calling any of the big FC or 10GbE players, please understand, I’m buying this for myself to test and learn and have only 3 nodes (Mac, Win 7, Linux). But I do have a commercial setup in my sights after this and any players that are willing to point me in the right direction and grow my engineering education will have my appreciation and my ear when it comes time to do the real build-out early next year.

REPLY – you ask about Infiniband in your opening subject line. Cal Digit sells “Super Share” and Accusys sells the same system – this is all Infiniband, it requires SAN management software, just like fibre channel (Fibre Jet, MetaSAN, or XSAN), and yes, it’s expensive, but it’s fast. So Infiniband is alive and well, along with 10G and Fibre Channel.

So just for fun, I was thinking of buying one of a few different IB setups I see on Ebay with various numbers of ports and matching cards. How can I not, I can get a switch and 3 10Gb/s cards for around $400 now a days and that’s low enough for just experimenting and learning.

REPLY – you can get 3 10G cards for $400 bucks ? From who? Chelsio, Myricom, ATTO, Small Tree ? This isn’t fun – you think this is fun ?

However, while the windows and linux drivers for Voltaire, Mellanox and other IB manufacturers are all available, I find nothing for OS-X.

REPLY – google OS X 10Gig ethernet. That’s how I found all of them.

My questions are:
1. What’s happened to Infiniband in general as a SAN topology, it seemed to have such promise and was cheaper than fibre but it seems impossible now to build a SAN with IB gear, I feel I’d really be on the fringe and with little support if I did. It’s seems like you can only do FC or 10Gbe (and I don’t understand the big push for 10GbE since FC is still faster and cheaper). Why is IB not as big as FC and has even diminished significantly in the last few years?

REPLY – IB is sold right now by Cal Digit as Super Share, and by Accusys. And its not “cheap”.

2. While there are still a few options for IB on Windows and Linux, the OS-X market is dead. Why? What happened? You used to be able to get IB cards for Macs, now I can’t find a single one.

REPLY – because you don’t know how to use a new invention called GOOGLE. I knew nothing about this, and all I did was google, and I found it. You can do the same, Mr. Smartie Pants.

3. When using IB or FC can you do simple file sharing (ala CIFS/SMB, NFS, AFP/IP and so on) over that topology (but at the higher speeds of IB/FC) so you don’t have to get into shared file systems? I realize it’s not going to be as fast as block transfers but it should still be faster than 1GbE. I know someone also testing a cheap IB setup off Ebay and he’s running IP over IB so he must be doing simple file sharing.

REPLY – boy, you sure like ebay. Every company on this forum is doing “simple file sharing”. Maybe you should read some of the threads. The entire market caters to OS X. And there are two mainstream manufacturers using IB.

And lastly, this is where surely I will get eviscerated:

REPLY – eviscerated – that’s a real big word !

4. Why are there no opensource shared filesystems that are linux, windows and OS-X compatible? There seems to be a few ways to get 2 out of 3 but OS-X seems to be a commercial proposition only. I’m almost tempted to run Linux in a VM under OS-X and share a drive back up to the host system just to test some open source options

REPLY – there are plenty of Linux shared storage systems on the market – they are commercially sold (think EditShare) and they are teriffic – and guess what – they use 10G ethernet as well as copper 1G ethernet. and there are plenty of others.

Mind you I am using this test setup to learn on my home systems. However it is my plan in the next 6 months to build out a commercial SAN for my company utilizing something along the lines of 8Gb FC and MetaSAN (or any easy to implement multi-OS cluster FS, hopefully without needing a MDC) between 2 servers (with DAS), 2 Smoke boxes and 16 Win 7 workstations. But baby steps first.

REPLY – for what reason – so you can look like a big shot to your company ? There are plenty of commercially available systems. MetaSAN is teriffic as a fibre system. You buy your seats of MetaSAN, a nice QLogic Sanbox Fibre Switch, a Fibre drive array (Active Storage, JMR, Sonnet or countless others) and you have a nice FC SAN, without your “homebrew” research. What will this prove ? Will your company lose money by not having the critical gear they need to MAKE MONEY NOW ? Why wait – BUY IT NOW. MetaSAN will work wonderfully in a fibre enviornment. Are you afraid to buy ATTO FC42ES cards ? Do you not like orange fibre cable ? It’s a nice aqua color if you use 10Gig .

If you research the market, (use google) you will find everything you are looking for.

Bob Zelin
Professional idiot

[David Gagne]

Sounds like fun. And by fun, I mean, “kids, don’t try this at work.”

If you’re gonna go for this, here’s what you could do:
1. Buy your crazy IB switch on eBay, if it comes with cards, use them on Windows/LInux hosts
2. Buy IB cards for the macs from someone who makes them for mac, like Cal Digit (not sure if they will sell individual cards, but you can try). Not sure it will be compatible, but hey, you’re a risk taker, right?
3. I guess you would have to share it out using SMB or NFS, or use MetaLan or StorNext…

That *could* work, but unless you’re just a crazy guy with time/money to burn, I’d go with a regular solution, either with 10GbE or Fiber.

[Bob Zelin]

you can just buy an Infiniband solution (Cal Digit Super Share or Accusys is this solution). Why reinvent the wheel. The price of Super Share is a very fair price, and is tested. Excuse my “boldness” my making this assumption, but I think that our friend Chris here wants to show his company how he can do this super fast wonderful IB shared storage solution for absolutely no money, and not buy an already existing package.

Well, Chris will find out what “research and development ” is all about, and why ALL of these companies charge a premium – because they have FIGURED OUT how to accomplish all of this, and you don’t give away this info for free.

Bob Zelin

[Bob Cov]

Wow, such a contrast in response compared to the other one. Much appreciated. Just found a bunch of great IB stuff on ebay dirt cheap. I too am looking at ways to get IB to work on OSX. Thanks for the tips.