[Chris Cantey]

Ok, here is the/a solution, for the later Googlers.

Ultimately we reached 1025 MB/s read and the same 850 MB/s write. We just had to tune the client sysctl.

We used the following:

net.inet.tcp.doautorcvbuf=0
net.inet.tcp.doautosndbuf=0
net.inet.tcp.sendspace=4194304
net.inet.tcp.recvspace=4194304
net.inet.tcp.maxseg_unacked=8
net.inet.tcp.delayed_ack=0
net.inet.tcp.win_scale_factor=7
net.inet.tcp.rfc1323=1

For our 2013 Mac Pro with ATTO Thunderlink NS 2201, these are what did the trick. The critical difference between this and what is often suggested (as well as what ATTO includes by default with their driver install) is that we did not set the kern.ipc.maxsockbuf to 8388608. We just left it off. It’s odd because the default is exactly this value, but hard setting it caused the read speeds to drop back to 20-40 MB/s.

The other settings seemed to have a smaller effect on the speed but they did inch it up. For example, maxseg_unacked=8 vs 32, gave about 80 more MB/s on the read. And the same for delayed_ack=0 vs 2 as well as the win_scale_factor=7 vs 3. And rfc1323 needed to be on, though this is the default.

The send and recv space definitely had an effect vs the default but these (or around these) are the commonly offered 10 gbe tuning values we’ve seen.

No switch changes were necessary and nothing on the server end. Jumbo frames were not necessary to reach these speeds. Our MTU is still 1500 end to end.

[Chris Cantey]

Thanks! And thanks for being frank.

Netatalk is the open source AFP server software that most of the off-the-shelf / custom built *nix based system are using.

Yes indeed these are the pains of custom solutions. We’ve had a lot of luck with them over the years so we keep at it. Overall, the pains of this custom build have been small compared to the gains. This one has been especially successful until this last turn with switching out the ATTO card for a Thunderbolt solution.

Thanks again for the input.

If anyone else has seen anything like this let me know. The excellent write and terrible read is what has been perplexing. Especially since we’re only seeing it after moving off of the PCI-based 10GbE card and/or to Yosemite. And only with the AFP protocol (vs raw network speed).

[Chris Cantey]

Hi Bob,

I did mention the other card, right at the top: ATTO FastFrame NS12

I work for a company called wonders, we’re in the U.S.

The switch is a Dell Powerconnect 8132F. Currently the workstations are connected via fiber with SFP+ LC SR modules. The servers connect with SFP+ copper. What else would be helpful to know about its config?

Our sever is a custom NAS using Netatalk for AFP.

I should clarify that I truly am just looking to see if anyone has encountered this one before. The hope is that someone has, they have some possible solutions, and they are available for hire (consulting, possibly remote assist).