Trouble Configuring 10GbE Network

[Bob Zelin]

oh come on Jonathan, you are not a systems administrator.

Your Mac Mini is not fast enough to handle all this bandwidth. You have the SanLink2 and the Lacie 8BIG on the same x4 Lane Buss of the Mac Mini. Mac Mini can’t handle the bandwidth that you are trying to force thru it. Its fine at 20 – 30 MB/sec, but when you get into multiple 10G client speeds, it’s choking. You said that you have most of the clients on SanLink2 boxes, and the older Mac Pro on a Small Tree 10G card – you can’t push that kind of bandwidth.

And your Lacie can’t do more than 800 – 900 MB/sec (it’s thunderbolt and only 8 drives, AND it’s on the same buss and the SanLink 2 that connects to your XS708E). And do you really know that your XS708E is running at Jumbo Frames ? There ain’t no setting for it on the switch. Should have spent the couple of extra bucks for the XS712T !

Get off your Mac Mini (make it a client computer) and switch to a Mac Pro Cylinder 6,1 – put the SanLink2 on one buss, and the Lacie on another buss, and you will see instant improved performance.

and exactly what are your sysctl.conf files, and are you connecting via smb or afp ?

You kids think that this stuff just plugs in and works – you have now learned that it doesn’t.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com

[Jonathan Loomis]

Thanks for you response Bob (however loaded). Swapping out the Mini for a Pro and putting the Sanlink and LaCie on separate buses was the next step I took in troubleshooting, but even though speeds are slightly better I’m still largely getting the same results with more than one client connected. I’ve tried over both afp and smb without much difference, and jumbo packets are pinging to the server so the switch is definitely sending them.

The part that boggles my mind though is that even when I connect clients over gigabit ethernet to the XS708e the same problem occurs: one gigabit client connected gives r/w of 100-110MB/s, but as soon as a second connects both of their speeds get choked up and they end of fighting over about 100MB/s of total bandwidth (one will be around 30-40MB/s with the other around 60-70MB/s). I think you can agree this is well below what the Mac Pro, Sanlink, and LaCie are all capable of, I’m just not exactly sure why it’s going on. sysctl.conf settings are below:

net.inet.tcp.doautorcvbuf=0
net.inet.tcp.doautosndbuf=0
kern.ipc.maxsockbuf=8388608
net.inet.tcp.sendspace=4000000
net.inet.tcp.recvspace=4000000
net.inet.tcp.maxseg_unacked=8
net.inet.tcp.delayed_ack=0
net.inet.tcp.win_scale_factor=7

Any helpful response is appreciated. And yes, this was definitely not my choice for a shared storage solution but I’m the guy who has to try and make it to work.

Thanks,
Jonathan

[Bob Zelin]

Hi –
the tuning file that you list was posted by Steve Modica some months ago. The 4000000 send and receive space is slightly wrong, but close enough. I build systems like this all the time – I just did a system for Good Great Grand in NY City with a Maxx Digital 8 bay thunderbolt array, and it’s working just fine – using the same promise SanLink2 that you are using.

I have seen the XS708E be problematic, which is why I now refuse to use the 708E and only use as a base model the XS712T, where I know that I can actually set things. Update your 708E firmware, and RESET the damm switch (I blame the switch at this moment, without knowing anything else). If you are using a Mac Pro 6,1 and have the Lacie on one buss, and the Promise on another buss (with your tuning file), you should have no issues. I assume you have tested the speed of your Lacie array, and that it is fast – correct ?

Are your clients 10GbE ? Promise boxes on your client computers ?
I have only seen the loading issue that you are seeing (30 – 40 MB/sec) when I use a Mac Mini as the server, not the Mac Pro 6,1.
I have done numerous installs like this with almost identical hardware (712T switch) and have not seen the loading like you are seeing UNLESS I use the Mac Mini as the server.

so when you play back your video on your clients, you are seeing stuttering video ? What editing software are you using ?

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com

[Jonathan Loomis]

Thanks again for the response Bob,

All of our clients are using Premiere CC 2014 and sometimes After Effects running through 10GbE Sanlink2’s. We updated the switch firmware which looks like it helped read speeds a little but writing from multiple clients still chokes up pretty badly. I had our 3 editors test out their Premiere workflow yesterday and things seemed functional for the most part. Today we had a couple additional editors throughout the studio connect over gigabit to transfer some files as well as use After Effects and that seemed to result in more stuttering playback and hangs in Premiere, so maybe we’re just limited to a very few number of clients?

We managed to isolate the issue to the storage by running r/w speed tests directly to the server’s local drive over the 10Gig network. Since the network had no trouble maintaining consistent speeds during these tests that leaves the connection to LaCie as the only possible bottleneck. After reaching out to their support the answer I received was basically that the 8Big was not intended or designed for file sharing at this capacity and that they don’t support it.

At this point I’m wondering if setting up something like SNS’s iSCSI server/client target/initiator software might help manage the random read/writes coming from multiple clients better? I don’t have any experience with it so any input anybody has on it would be very useful. Other than that, it sounds like replacing the RAID array with something like a Maxx Digital is the only other thing I think we could do.

– Jonathan

[Bob Zelin]

I will contact you.

Bob Zelin

Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com

[Olivier Jean]

Hi Jonathan,

How did you solve your issue in the end ??

Regards
Olivier Jean
Video Sales Consultant
Apple Certified Trainer Final Cut Pro 7
Powermedia Systems
Sydney Australia

[Jonathan Loomis]

Hi Olivier,

After troubleshooting/ testing every other component in the signal chain (NIC’s, client machines, switches, cables, server, software, etc.) we ended up replacing the LaCie 8Big with a Maxx Digital 48TB ThunderRAID 2 and haven’t had any issues since. The RAID controller on the LaCie for whatever reason just can’t negotiate traffic properly from multiple clients. In the words of LaCie’s customer support the 8Big units (and probably their others too) weren’t designed to do this and therefore they don’t offer support for this kind of networked setup. Thanks again to Ryan and Bob over at Maxx Digital for their help getting us a better rig.

Final working signal chain ended up being:

Thunderraid2 -> Mac Pro server -> Promise SANlink2 -> Netgear XS712T -> Promise SANlink2’s -> 4 – iMac/ Mac Pro Premiere CC 2014 clients

Cheers,
Jonathan

[Craig Ricker]

Our company has a Synology Rackstation RS3614xs, I was tempted to make a move from internal raid 10’s to everyone (3 mac pro edit suites) editing directly from the NAS in a raid 6 setup. Its currently using 8 bays on WD 6TB Red NAS drives.

Is it my understanding that most NAS systems will bottleneck on random access read/writes because a NAS isn’t optimised for such use? Or is it just some NAS systems, where they didn’t build the product with multiple editing video streams in mind?

Mac Pro 2.4Ghz 8 core, 24GB RAM, GTX 670

[Jonathan Loomis]

Hi Craig,

There are many NAS systems suitable for video editing but I would not recommend the 8Big as the storage component in a NAS. As I understand it with most video you’ll primarily be concerned with sequential read/ write speeds rather than random r/w you’d see with a mail server or database.

If your needs dictate a shared storage solution make sure your throughput requirements are met by both the Synology RAID hardware and your network. A NAS will also require more tuning and/ or support hours than editing off of internal RAIDs so it’s well worth it to find an expert you can reach out to if you run into issues. We currently have 2-4 Premiere editors at a time working successfully in H.264 off of a Maxx Digital Thunderraid2 and Mac Pro server on a dedicated 10GbE network.

– Jonathan

[Adam Sparks]

Hi Jonathan

How would you rate this setup in terms of its simplicity to administer? Using the same products and tuning file, is this relatively set and forget for someone with an average handle on networking?

I see Bobs condescending comment that “You kids think that this stuff just plugs in and works – you have now learned that it doesn’t.”

But it seems to me from wading through these forums that if you are capable of setting up a robust 1G network, using the right equipment in the right configuration should not require a terribly steep learning curve to adopt a 10GbE workflow

What would you say to that?