Is link aggregation the solution?

[Jesse, Dijifi]

Thanks Bob.

You are right, although I have a very unique situation in which (according to the manufacturer of my capture machine and accompanying software) I must first capture these huge files to an internal drive of what I will call the ‘ingest’ station (of which there are two). The capture of 8mm/16mm film to digital files will apparently not work if I attempt to capture directly to shared storage or anything not directly connected to the motherboard of this ingest system (the reasoning I don’t completely understand, but the manufacturer assures me it is so).

So, I at least have to have internal arrays for the initial capture. And since I only need to edit on one station, and in the past have tolerated the time it takes to transfer from one station to the other, I just went with the more affordable G-Speed eS array which achieves up to 500-600 MB/s on the local system. I figured if the network were fast enough it could talk to the other two machines at at least over 100 MB/s since they are RAID 0.

So this direct connect method is very interesting to me. Though I don’t know if it can direct connect two stations to a third station, rather than one to one.

[Alex Gerulaitis]

[Jesse, DiJiFi] “So this direct connect method is very interesting to me. Though I don’t know if it can direct connect two stations to a third station, rather than one to one.”

You can with either two 10GbE cards or one dual-port 10GbE card in the edit station.

This will also likely require a static IP setup in each card (no DHCP server in the segment) – which isn’t anything to worry about – fairly straightforward.

Alex (DV411)

[Jesse, Dijifi]

Oh cool! Okay, I will look into this.

Alex, do you think someone with pretty basic knowledge of networking could set this up without too many problems? I build our computers from parts, so I know a fair amount, but am no expert in networking.

Again, infinite thanks for your input on this. All of you.

[Alex Gerulaitis]

[Jesse, DiJiFi] “Alex, do you think someone with pretty basic knowledge of networking could set this up without too many problems?”

Yes I do think so. The only thing I am not sure about is how to prioritize the 10GbT link over the standard one when you do those file transfers – but with Steve on our side, pretty sure we can figure this out. 🙂

Alex (DV411)

[Steve Modica]

Routing is handled automatically depending on how you do the mounts.
The ports cannot be on the same subnet (or the kernel routing table will get confused).

When you mount the other machine, use “connect to server” and give the 10Gb address. That will automagically route over the 10Gb. Then you can drag the mount down to the tool bar to save it.

Steve Modica
CTO, Small Tree Communications

[Alex Gerulaitis]

These sound like OSX instructions – I think they can be adapted to Windows. Thanks Steve!

Alex (DV411)

[Steve Modica]

Right! It’s the same deal tho. You connect to the share using the 10Gb IP address. The routing issues probably apply. If you have two ports on the same subnet, you are essentially lying to the OS. Since they are on the same subnet, the OS *thinks* that both ports can hear the same packets, but in real life, they can’t. So it will send packets out one port for *both* ip addresses. (old problem)

Steve Modica
CTO, Small Tree Communications

[Andrew Richards]

[Alex Gerulaitis] ”
Great info – thanks Steve. I had no idea TCP had that much of a computational overhead. Are there ways to reduce it and speed up 10GigE to get it closer to its 10Gbs ceiling?
“

Aside: has TOE ever been attempted on OS X? I know on Linux it requires kernel extensions as well as drivers…

Best,
Andy Richards

VP of Product Development
Keeper Technology

[Steve Modica]

[Andrew Richards] “Aside: has TOE ever been attempted on OS X? I know on Linux it requires kernel extensions as well as drivers..”

Not really. Neterion tried and had a large hardware abstraction layer. They kept telling me it was done, but the driver never released and was always “alpha” and you had to ask them to mail it to you. They never mailed it to me 🙂

Chelsio did a slow path driver (no TOE), and claimed they would have TOE later. From my understanding, the upper management of software within apple has no intention of putting in the kernel TCP stack hooks to let that happen. (I’ll explain that in a sec)

We tried to do it with Intel’s “quickdata” dma offload engine. That was an exciting project. We got the design all completed and then Intel decided not to move forward with the chip, so we were screwed. But even in that case, we needed some Apple help to make that work.

So what’s needed??
Ultimately, when you transmit data out of the machine, the OS puts the data into mbufs. It has to keep these around in case they get lost and have to be retransmitted. however you are writing to your socket buffer and could care less that the OS needs to keep those buffers. You’ll happily overwrite them. So the OS has to copy them off to a safe place in the kernel. So all your data is memcopied before it hits the network. This uses a *lot* of cpu. Each word has to go through a ld/sd operation.

TOE and quickdata were supposed to subvert this. However to do that in the BSD kernel stack, there’s a check to see if an offload path exists. If so, it’s called rather than the normal memcopy. Apple does not have this check and has not wanted to put it in. Hence, no TOE.

(Historical note: Dave Miller hated TOE and refused to let it into the Linux stack. So those guys all had to release kernel patches to support their stuff. That was a huge hassle and was yet another reason TOE failed)

Steve Modica
CTO, Small Tree Communications

[Jesse, Dijifi]

Okay, at some point this summer I am giving this a try. Thank you again for all the helpful information. I really appreciate it guys.