Promise SanLink2

[Kieran Steele]

Boris,

I get the same results as you with aja, 10.9.4. First run after a boot ;again with cache off; real speeds, each test after that, some sort of super fast cache speeds. I too switched to black magic disk test when I encountered that issue. I’m not sure if my tcp tuning as above is having an effect. What was your before and after figures, and did you notice increased speed on 1gbe as well as 10gbe (presume not), and AFP as well as smb, or only smb. I presume it’s system wide and across all nics no matter how they are connected.

[Baule Alessandro]

Hello,

I buy 9 sanlink2 10baseT for work with Netgear XS712T Switch and a FreeNas Server with dual 10Gbe ports RJ45.
The Server is OK, transfers about 1`gbps is ok, the Macs for Video edition with Adobe Premiere works great, is possible edit and play in file videos together.

Now we make the migration for 10gbe and nothing more works, don’t is possible play in the files in the Premiere, the video freeze, dont together with sound, delay all.

Please, help with this support, are 9 Sanlink adapters that dont work in 10gbe.

[Boris Tsipenyuk]

I have the same configuration here are some important tips:

Make sure all switch ports are set to jumbo frames (not on by default)

Make sure network interface on OS on client and server is set to jumbo frames

The rest of these worked well for me but results may vary!

On the client Mac enable the following settings (running via Sudo don’t survive reboot for testing)

sudo sysctl -w net.inet.tcp.doautorcvbuf=0
sudo sysctl -w net.inet.tcp.doautosndbuf=0
sudo sysctl -w kern.ipc.maxsockbuf=4194304
sudo sysctl -w net.inet.tcp.maxseg_unacked=32
sudo sysctl -w net.inet.tcp.delayed_ack=2
sudo sysctl -w net.inet.tcp.win_scale_factor=7
sudo sysctl -w net.inet.tcp.rfc1323=1
sudo sysctl -w net.inet.tcp.sendspace=1048576
sudo sysctl -w net.inet.tcp.recvspace=1048576

I am unclear if you are using a Mac or Windows server. What I have found on windows servers is I disabled ALL tcp offload options on the interfaces, but make sure receive side scaling is enabled.

Also if a windows server in local security policy make sure to disable samba digital signing (I’m mobile so I can’t look up exactly what this setting is called)

[Baule Alessandro]

The Promise dont have Jumbo Frames (mtu 9000)
My server is a Freenas – BSD Based
The server is set mtu 9000

i think that mac make a cache for play files, and when the video freeze the mac dont access the server, i see access in the “server disk” like zero mb/sec.

Have a special config for premiere works in the 10gbe?

thanks

[Baule Alessandro]

The Promise dont have Jumbo Frames (mtu 9000)
My server is a Freenas – BSD Based
The server is set mtu 9000

i think that mac make a cache for play files, and when the video freeze the mac dont access the server, i see access in the “server disk” like zero mb/sec.

Have a special config for premiere works in the 10gbe?

thanks

PS we use a AFP protocoll

[Steve Modica]

We get contacted a lot for things like this.

People take some server hardware, install redhat, freebsd or windows, and then expect to hook it all up with 10Gb and have it go fast. It usually won’t for a number of reasons.

Typical operating systems are not tuned to deal with 10Gb ports very well. Their window sizes are not large enough in some cases. On others, they allocate a ton of memory for each port, but it’s not optimized for single stream video editing.

I have a feeling the Netgear switch does not have flow control on. Most switches ship with it off, and in standard core/edge switch setups, this is preferred. However this is not a core/edge setup. This is a single layer setup with a large number of systems pulling data from the server. Symmetric flow control is necessary.

Promise supports jumbo frames. However Apple’s system preferences tool has a bug. It will not allow jumbo frames if you set the speed of the device to 10Gb. Set it to Auto and it works. This is fixed in the next release.

Steve

Steve Modica
CTO, Small Tree Communications

[Baule Alessandro]

I think that the problem is more with macosx and 10 gbe particular.
The migration From 100 to 1000 gbps was more easy, but now for 10gbe have very complications.
The line to way is flow control, sysctl in macosx, and mtu 9000 to all adapters and switch?
Have a secrets from macosx to all work ?
Definitely 10gbe don’t is plug and play.
Thank you very much for your attention .

[Simon Blackledge]

Anyone noticed the if your sharing out storage over 10Gig the Server Macs ram gets totally gobbled up?

Never freed up even when clients disconnected. Even after a weekend of no connections.

Is this to do with the settings in sysctl.conf ?

keep having to run sudo purge.

Never had this over 1Gig with other machines. Should the server be running OSX Server to cope with this?

Or is osx just pants at flushing cache with 10Gbit ?

S

[Chris Duffy]

This is a fact of life 🙂
Anytime you have local storage and “share” it out on a Mac,
Mac OS will take all the memory that is not being used and allocate
it for it’s buffer cache….always been that way….does not matter
what type of networking you are using…..Of course Mac OS will steal
it back for an application or program that needs it for something else….
so on most Mac’s that share storage you will always notice that
most of the memory is being used 😉
and yes, “purge” command (if you have the developer tools installed)
will get it back temporarily but give it some time and it will
be gone again 🙂

What I have found over the years is people forget to tune the
kernel’s filesystem buffer that it can use in working with files.
Called kern.maxnbuf (maximum size of the filesystem buffer).
Mac OS usually defaults this kernel
parameter to 16384 but it does have a formula it uses to set
it depending on the amount of physical memory
but a file server with lots of memory needs more of this to work
optimally…… So if you want to help Mac OS server do it’s
job better, jack up this tunable….. You can find some articles
on the web on what to set it to…..Start with 262144 and for
most systems this should be enough…..maybe
go up to 524288 if you have LOTS of memory…..
should not need more then this even with
large memory Mac’s.

[Philipp Meier]

Would love to re-visit this topic. Also using the Promise SanLink2 and getting decent performance connecting to a Windows Server 2012 R2 with a 36 drive RAID 60. Internally the RAID gets 3,000MB/s read and write.

A Mac Pro 2013 with a Promise SanLink 2 currently gets 630MB/s writes and 840MB/s reads with the following systlc config:

net.inet.tcp.doautorcvbuf=0
net.inet.tcp.doautosndbuf=0
kern.ipc.maxsockbuf=8388608
net.inet.tcp.sendspace=4000000
net.inet.tcp.recvspace=4000000
net.inet.tcp.maxseg_unacked=8
net.inet.tcp.delayed_ack=0
net.inet.tcp.win_scale_factor=7

Before that I used these:

net.inet.tcp.doautorcvbuf=0
net.inet.tcp.doautosndbuf=0
kern.ipc.maxsockbuf=8388608
net.inet.tcp.sendspace=4194304
net.inet.tcp.recvspace=4194304
net.inet.tcp.maxseg_unacked=32
net.inet.tcp.delayed_ack=2
net.inet.tcp.win_scale_factor=7

That got us to 632MB/s write and 828MB/s read. Seems like decreasing the unpacked value to 8 and setting ack value to 0 squeezed out just a little bit more. Possible that it was just a fluke, though.

Anyways, the performance isn’t bad but the write are much slower than the reads. Again the RAID is not the bottleneck as it can pull 3000MB/s writes. Wondering if there are any additional settings on either the OSX or windows side that can be tweaked to improve the speed?

SMB signing is already turned off and Netbios over TCP is turned on on the Windows NIC. Jumbo frames enabled on Client, Server, and Switch. Switch is a Netgear M7100-M24X.