-
Fibre Channel Setup Debauchery
Dear Everyone,
After writing something that was vague, not to mention in the wrong place, and haven had a cheerful reply from a certain grumpy (but highly respected!) editor, I have cleaned this up to higher standards and am currently seeking a take-two in whatever advice can be derived from this forum.
Preface: there is a small film studio being started up in eastern Europe, and I was invited to make an “inexpensive” (Yes, a very dangerous word to mention) and simple Fibre Channel network to accommodate their video demands. Little did I know there were some inherent flaws in my setup, and I had some incorrect conceptions about the nature of FC and SANs…
Hardware:
– Mac Pro, 8 core 2.26ghz, 8gb RAM, 2tb internal HD, ATTO Celerity 42ES Fibre Channel card, 10.6.newest client
– PowerMac G5, 2.0 dual, PCI-X, 4.5gb RAM, 500gb mirrored RAID for system, ATTO Celerity 42XS fibre card (4gb, 2 channel) running 10.5.8 serverRunning on the G5 are two extra 4-port eSATA cards, that connect to two rack-mounted eSATA RAID arrays.
We can’t do local storage on the Mac Pro, so it was decided that the G5 would be the server.
(To elaborate, the Mac Pro is in the studio, and the studio needs to have as minimal noise as possible. Mac Pro’s usually don’t generate too much noise, and we have it in a sound-dampening box, whereas the G5 and it’s loud RAID arrays get to make as big of a racket as they want, since they’re located in another room)
In an ideal world, and what made conceptual sense in my head, was that the G5 would host the SATA drives, the SATA drives would appear on the desktop of the Mac Pro, and that SAN software was not needed, because there was only really one computer accessing the data on the G5. Since the G5 would simply be acting as the server, it seemed like we wouldn’t need SAN software…
Right?
Wrong!
As everyone obviously knows, Fibre Channel has this entire target and initiator relationship, where most HBA cards are initiators, and the Fibre Channel arrays are the targets. What’s awesome is that even though certain Fibre Channel HBA vendors advertise that their cards support target mode, that their cards even have an API that goes with it, actually changing to and enabling this mode is located in an entirely different ballpark! I have been personally told that there’s lots of “red tape” and IP (an acronym for “intellectual property”) at stake, and unless we’re some big company or OEM, this software or functionality isn’t available to “end users” like me.
No matter, though– it’s not like I’m a programmer or anything. It’s not like I derive any joy from playing with registers and micro-op crackers.
BUT! It brings a good question to the table– hypothetically, if I did get one of these HBA’s to switch from initiator to target mode, say the HBA in my G5, would all drives connected to the G5’s system bus magically appear on the Mac Pro?
And here is another question, just a simple question about Fibre Channel that I should have known before I got myself into this: if I hypothetically took the Mac Pro with a Fibre Channel HBA card and connected it to a Fibre Channel RAID block (eg Terracube, or something nice and warm and fuzzy for storing delicious data on), would the storage device appear like a regular disk would? I mean, in the context of Mac OS X, would I see the volumes in disk utility?
So continuing on, just a few more questions for you all to ponder and debaucher,
We simply want a setup where we can store a bunch of video files in a location separate from where they would presumably be edited– hence, the Mac Pro in the studio, storage in a separate room. And it needs to be fast:
As I “understand”, something such as 10-bit uncompressed video is too fat to stream over a Gigabit Ethernet. It appears that it might have been possible to stream it over a 10-gig-Ethernet or Bob Zelin’s 4-gig-Ethernet solution, but that is unfortunately out of the question at the moment, as we have the Fibre Channel cards, and want to work with what we have.
The simplest solution seems to be a Fibre Channel RAID storage unit. This seems like the best solution, the correct, way to develop what already exists.
So… please– correct me where I’m wrong!!
Please and thank you,
Spencer Homick
-
8 Replies
-
Spencer,
I’ve provided some kind of responses below —
(you say..)
Preface: there is a small film studio being started up in eastern Europe, and I was invited to make an “inexpensive” (Yes, a very dangerous word to mention) and simple Fibre Channel network to accommodate their video demands. Little did I know there were some inherent flaws in my setup, and I had some incorrect conceptions about the nature of FC and SANs…(my reply..)
I already see a couple of problems with the statement. Fiber Channel, Inexpensive, and Simple ….well frankly, Fiber channel never has been any of the above.The last time I talked to a customer who wanted to do a fibre channel san on their 4 Mac clients (last week..). They were getting quotes of 20K just to get it in the door…..
I gave them a 9K quote to use an Ethernet based solution, and they will be lasting a lot longer then that Fibre Channel SAN they looked at!
(you say…)
Running on the G5 are two extra 4-port eSATA cards, that connect to two rack-mounted eSATA RAID arrays.We can’t do local storage on the Mac Pro, so it was decided that the G5 would be the server.
(To elaborate, the Mac Pro is in the studio, and the studio needs to have as minimal noise as possible. Mac Pro’s usually don’t generate too much noise, and we have it in a sound-dampening box, whereas the G5 and it’s loud RAID arrays get to make as big of a racket as they want, since they’re located in another room)
(my reply..)
They do know that the G5 is going to offer much less performance then the Mac Pro don’t they? — it’s a completely different hardware architecture, not to mention PCI X…..not PCI Express. Even using a Mac Pro Quad Core would be better then using a G5….Perhaps they are okay with the performance of the G5 vs the Mac Pro…(you say..)
In an ideal world, and what made conceptual sense in my head, was that the G5 would host the SATA drives, the SATA drives would appear on the desktop of the Mac Pro, and that SAN software was not needed, because there was only really one computer accessing the data on the G5. Since the G5 would simply be acting as the server, it seemed like we wouldn’t need SAN software…So continuing on, just a few more questions for you all to ponder and debaucher,
Yes — you can successfully deploy a Mac Server, and put all your users into it, and use the server for the point of access, taking advantage of the storage….(this is how these Ethernet based configurations work…without SAN software, and over AFP…)
As it is with Fibre Channel, I believe in general (and someone can correct me …) is that you are required to put a software license on every person’s machine accessing the storage…because Fibre Channel is not traditionally deployed behind a “server” as much as it’s storage, hanging of switches….
(you say..)
We simply want a setup where we can store a bunch of video files in a location separate from where they would presumably be edited– hence, the Mac Pro in the studio, storage in a separate room. And it needs to be fast:As I “understand”, something such as 10-bit uncompressed video is too fat to stream over a Gigabit Ethernet. It appears that it might have been possible to stream it over a 10-gig-Ethernet or Bob Zelin’s 4-gig-Ethernet solution, but that is unfortunately out of the question at the moment, as we have the Fibre Channel cards, and want to work with what we have.
(my reply..)
Gigabit speeds on a Mac Client out can be achieved at 70-80MB/sec
10Gb Speeds on a Mac Client *10.5.8 can sustain 450-500 at the server with multiple incoming connections out per client (per client will run around 250-300MB/sec…)(you say..)
The simplest solution seems to be a Fibre Channel RAID storage unit. This seems like the best solution, the correct, way to develop what already exists.(my reply..)
No…the Simple way, would be doing this with Ethernet, where you don’t have all the nasty overhead of san software, read / write permissions, etc….Matt Geier
Small Tree
https://forums.creativecow.net/smalltree -
No, Here is the simple solution:
Buy a fibre channel RAID and a 60 meter fibre cable. Stick the RAID in your rack in the other room, run the cable to your onsite mac pro and be done.
OR
But a Geffen extender, or something like that, put a monitor and keyboard in the studio, control your MP remotely and plug the esata boxes into your MP.
and
put your G5 in the recycle bin.
-jw
-
Hi Guys,
Thanks for your help in this. I’ll respond to both of you respectively:
(MATT) I already see a couple of problems with the statement. Fiber Channel, Inexpensive, and Simple ….well frankly, Fiber channel never has been any of the above.
(ME) I wholeheartedly agree!
(MATT): They do know that the G5 is going to offer much less performance then the Mac Pro don’t they? — it’s a completely different hardware architecture, not to mention PCI X…..not PCI Express. Even using a Mac Pro Quad Core would be better then using a G5….Perhaps they are okay with the performance of the G5 vs the Mac Pro…
(ME): We are very aware of this. In this hypothetical context, the G5 would be acting as the link between the eSATA RAID arrays and the Mac Pro. All the video processing would be done on the Mac Pro, so in retrospect, I wasn’t worried about the capabilities of the G5 to process video. Even if PCI-X is comparatively slower than PCIe, does it really make that huge a different if all it’s doing is transferring information from one or two eSATA RAIDS by way of one or two 4-port PCI-X cards to the ATTO card and then to the Mac Pro? I guess my assumption is that a lot of processor work is being done on the HBA and eSATA cards, and that PCI-X is fast enough to accommodate 10-bit uncompressed and upwards.
Or am I wrong, horribly, wrong?
(MATT): As it is with Fibre Channel, I believe in general (and someone can correct me …) is that you are required to put a software license on every person’s machine accessing the storage…because Fibre Channel is not traditionally deployed behind a “server” as much as it’s storage, hanging of switches….
(ME): I simply wonder, though, if the HBA was in target mode, and the G5 was presumably in a position that it was only doing server things like managing VPN and web connections, processing data that primarily resides on the internal hard disks, can I get by without having to install and license both systems with SAN software?
I think my response is just getting more and more whacked out, but I do feel like there are some hypothetical questions I am looking for answers to, so let me elaborate:
When you have a SAN RAID (target) and multiple initiators (workstations) accessing the single target, it is obvious that you have to have SAN software to co-ordinate the file read/writes so that catastrophe doesn’t happen.
So I am wondering, is catastrophe (corruption of data and filesystems caused by reading and writing of the same blocks of data) caused by the user layer, by general actions in either the filesystem / system layer, or even the lower system layers? If you had four computers connected to one target, with the target mounted on all four computers, and ONLY ONE computer was actually making any filesystem reads or writes (the other three plebeians are diligently keeping their cursors away from the mounted volumes), will the RAID filesystem still break itself, or will it stay intact?
(So, you see, if I have a G5 that is able to act as a target, any recognized drives on the eSATA PCI cards are routed both to the G5’s system (and are mounted), as well as to the Celerity cards and later to the Mac Pro, would the G5 corrupt the Fibre Channel RAID naturally, or could I get by through un-mounting the RAID volumes? It’s just a hypothetical question that would help me understand the internals of this networked storage system).
(MATT): No…the Simple way, would be doing this with Ethernet, where you don’t have all the nasty overhead of san software, read / write permissions, etc….
(ME): I like your thinking, and I do admit that given how magical Ethernet is, and know what I know now, this seems very, very nacho…
(JORDAN): Buy a fibre channel RAID and a 60 meter fibre cable. Stick the RAID in your rack in the other room, run the cable to your onsite mac pro and be done.
(ME): In the words of Guy Kawasaki commenting on how he wrote emails circa 1995, “Curt but complete”. Sounds solid, simple, and straightforward!
(JORDAN): But a Geffen extender, or something like that, put a monitor and keyboard in the studio, control your MP remotely and plug the esata boxes into your MP.
(ME): I might just encounter that setup in my dreams tonight!
(JORDAN): put your G5 in the recycle bin.
(ME): Awww, now you’re being mean : ) Isn’t going to happen, because we do need a VPN server, and we wish to have hot, steamy VNC and FTP access from abroad.
Thanks a HEAP!
Spencer
-
(ME): We are very aware of this. In this hypothetical context, the G5 would be acting as the link between the eSATA RAID arrays and the Mac Pro. All the video processing would be done on the Mac Pro, so in retrospect, I wasn’t worried about the capabilities of the G5 to process video. Even if PCI-X is comparatively slower than PCIe, does it really make that huge a different if all it’s doing is transferring information from one or two eSATA RAIDS by way of one or two 4-port PCI-X cards to the ATTO card and then to the Mac Pro? I guess my assumption is that a lot of processor work is being done on the HBA and eSATA cards, and that PCI-X is fast enough to accommodate 10-bit uncompressed and upwards.
Or am I wrong, horribly, wrong?
Here’s what you should really do, I think you are over doing your question.
(that’s not meant as a mean thing……just think about this for one second…)
Put the Mac Pro as the Server, Use the G5 with VPN, Final Cut (or whatever application) Put your storage behind the Mac Pro, and configure it for sharing across the network.
(as long as your application runs over a network, you can have it connect to things across the network…makes sense)
Connect Ethernet from G5 to Mac Pro and Share the storage from the server with the Mac Pro doing all the work, Server, Storage Server, Client, VPN, etc (Apple makes very good hardware as we all know…)
Find out if you can make that work with what you want to do. It might require you to continue further testing to see what kind of results you can achieve.
You are doing a couple of things by changing the G5 and the Mac Pro around.
1) Mac Pro has PCI Express…
2) Mac Pro has faster architecture then the G5 (and is also Intel based..)
3) It has better CPU’s for more Server activity…
4) It either has it now, or should have more memory then the G5 does now….Let me know ..
About the Fiber Channel related questions, I’m not an expert there….plenty of others here are however. I’ll defer to anyone else that has more knowledge on that type of conversation..
Matt G.
-
[Jordan Woods] “But a Geffen extender, or something like that, put a monitor and keyboard in the studio, control your MP remotely and plug the esata boxes into your MP. “
I’ll second this — this seems to be by far the simplest and most reliable solution for your current needs. You didn’t mention if there were any VTRs, but if so, everything can go in the machine room.
If you go this route, don’t just extend the keyboard, video and monitor — I’d also look into the Gefen USB and Firewire extenders to accommodate the inevitable sneakernet flash drives or hard drives.
Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog – What I’m thinking when my workstation’s thinking
Creative Cow Forum Host: Live & Stage Events -
I assume you will have to capture video from the floor too. In this situation the Geffen extender will not allow you access to things like audio/video I/O from something like a Kona Card.
I understand your situation. You need your fast video import station on the floor and the fast drive off the floor. I am trying to do this as well. If a Atto Celerity card was attached to my older MacPro running OSX 10.5 Server, it would basically be a super Fibre Bridge/Server. You could share all your drives over Fibre, Firewire, Fast ESATA, Etc.
Seems like you need the Atto Celerity Target Mode Driver. This would allow you to connect 2 Macs together and mount the drives of the Mac off the floor, on the floor Mac.
From my research, I think they reserve this driver for OEM clients so that they can eliminate people with low budgets from making fibre servers from their old Mac for very low cost. Maybe someone has a Target Mode Driver and would be willing to share.
Atto advertises that they offers a Target Mode Developer’s Kit and Target Mode Support without any disclaimers, but when you talk to them, they say that they only let large clients have this. It is very misleading. False advertising on Atto’s part. For them to say that the cards support Target Mode, when they don’t is a big deal.
Lets see if anyone can help with a Target Mode Driver for these cards, because others would greatly benefit from this.
Regards.
Caston -
Hi-
I think the route we are going to take is to simply locate a Fibre Channel RAID storage system, and connect it directly to the Mac Pro. The G5 is simply going to be used for VPN, as well as archival through Ethernet.
In reply to doing the reversal, it’s not possible, because we only have an AJA card for the Mac Pro, and not the G5.
I guess my final question is, say I purchase a Fibre Channel RAID, and I plug it directly into the Mac Pro, will the drives appear in Disk Utility?
Thanks and happy new year!
Spencer -
in a typical scenario, the fibre channel raid should show up as at least one un-initialized disk in disk utility. From there it is up to you to put the file system on it… HFS+, etc…
With a decent length fibre cable you should be good to go, and plenty of speed. (at least 500MB/s)
-Jordan
Senior Systems Engineer
Active Storage Labs
Los Angeles, CA
Reply to this Discussion! Login or Sign Up