Promise SanLink2

[Elvin Jasarevic]

This is our tests using the two Sonnet box and DDP. What do you think about this?

We also found that SanLink2 would not play nicely with ProTools. Cut cut story short, when you have 2-3 hours Video/Audio together, audio will be off few seconds. Using the Sonnet box solved our issue.

[Barry Sharp]

I have just ordered two PROMISE SANLink2 Thunderbolt 2 to 10 Gb/s Ethernet Adapters. My intent is to use these for two of my office Apple MP6,1 systems.

I will also use them for testing at first using my home Apple Late 2015 iMac17,1 27″ 5K Retina (32GB RAM) and my Mid 2012 Apple MBP9,2. (16GB RAM) The iMac has Thunderbolt 2 and the MBP has Thunderbolt 1.

The office MP6,1 systems have Thunderbolt 2 and 64GB RAM.

All the Macs are running macOS Sierra.

I notice on the MBP’s Network panel “Advanced…” I cannot set MTU above 1500, whereas on the iMac I can set MTU to 9000. I’ve not checked this on the MP6,1 system at this time.

The reason for trying out these SANLink2 devices is to see if the data transfer between our two office MP6,1 systems is better than using a simple Thunderbolt Bridge. This technique is very cost effective as it requires just a single Thunderbolt cable. However, we see varying issues over time using the Apple’s SMB and AFP. Sometimes SMB is better and sometimes AFP is better. It seems to vary with the macOS version we run. Currently running Sierra we find AFP is much better than SMB whereas pre Sierra we found SMB was better. The other issue we see with using the tHunderbolt Bridge is that the i/o rate is VERY choppy, and at this time is very bad using SMB vs. using AFP. I attach a graph (hopefully I’ve done this correctly) showing what I’m seeing using AFP. The orange is for writing from my MPB to the iMac and the blue is for reading from the iMac to the MBP.

So the writing goes along and improves in two distinct steps which I don’t understand (that is, why the sudden steps). The reading shows an initial ramp up and then continues to be very ‘choppy’ with lots of stalls.

I’m hoping the SANLink2 will address these issues and smooth out the data transfers and consequently provide a better overall average i/o rate.

Question: Does the SANLink2 require special driver code that needs to be installed, or is it a plug&play setup much like for the Sonnet Adapter which I’m told does not require driver code to be installed ?

Thanks for any attention to this issue(s).

[Chris Duffy]

Well, Well, if it is not Barry Sharp out west?
Duffy here…….
I am sure many folks will respond to you but I’ll give you a few
ideas and of course, if you want it fixed in leas then 10 minutes, why waste time here,
just let me teamview/screen-share with you on that mac and we can
resolve quickly ?

-Yes the Promise SANLINK2 needs a driver…..
go to promise’s website, sanlink2, download their 1.7.4 driver,
install and reboot.

-In the %$^% Apple Network Preferences manual,
if you want to use jumbo/9000 byte frames, first
select advanced,
manual instead of automatic,
change to autoselect
then you can change mtu to 9000..
save and so forth.

-Samba huh? lol known bug with samba on mac OS El Capitan (and Sierra) with
“signing”…….use afp for now and if you really want to use samba, there is
a workaround you can do on the macs.

-If you are using a switch to connect 10Gbe you will
need to configure jumbo frames and symmetric flow-control.

let me know how it goes or if you want me to fix it ?

Duffy

[Barry Sharp]

So my attached graph did not make it into my posting!

Thanks for the reply and the advice about the need to download software driver code from Promise site. I’ll do that ahead of time.

I’ll try attaching my graph again here.

This graph is me using a Thunderbolt Bridge between my home MBP and my home iMac.

The i/o traffic writing and reading across my Thunderbolt Bridge using my MBP as Server and my iMac as Client. I performed the i/o from the MBP. This was using AFP which performs better than does SMB under Sierra 10.12.1 (build 16B2584a). Even so, you can see how the writes run fairly smoothly but ramp up over the course of writing in two distinct steps (the orange graph). For reading from the iMac (the blues graph) you can see how choppy the transfer is, and also note how it ramps up initially before its gets going.

The peak write (orange) achieved here was around 800 MBytes/sec and the peak read (blue) tops out at around 1350 MBytes/sec. However the average rate is much much lower because of all the writing step-ups and the stops & starts in the reading transfer.

So I’m hoping/expecting the SANLink2 will avoid these writing steps and the ‘choppy’ reads. Should this expectation be real I ask ?

BTW…. on my MBP I cannot set MTU to 9000 no matter what I do. On the iMac I can set MTU to 9000. Why is this ?

Can I use a command in Terminal on the MBP to force/setup MTU=9000 ? If so, what would this commend be ?

[Barry Sharp]

Chris:

Yes, sorry for not responding to you saying…”well, well etc”. I’m now retired from HPC and do all the IT at local office for my son and my consulting work. We are an all Mac setup with MP6,1 12core and beastie iMacs.

We’ve been getting along kind of AOK using a simple Thunderbolt Bridge between two of our MP6,1 systems and for the most part it works except that Apple keeps tuning/tweaking/braking the networking software layers (SMB and/or AFP) moving from one OS version to the next. Very annoying. We had been using SMB for the TB Bridge prior to El Capitan and Sierra. We now find that AFP provides a much much better i/o rate. It means with each macOS version/update we have to test the TB Bridge i/o to make sure there’s been no regression. The TB Bridge certainly is a low cost solution and can move data at well over 1000 MBytes/sec in spurts, but cannot maintain that over an entire file transfer. It’s choppy and this results in the overall average i/o rate to be a lot lower. This 1000 MBytes/sec is when the MP6,1 server system (which has 64GB RAM) has most, if not all, of the data in its kernel buffer cache which can deliver well over 5000 MBytes/sec. This 5000 MBytes/sec obviously overloads the Thunderbolt-2 by a factor of 2, but does show as much as 1500 MBytes/sec at times.

With the use of the SANLink2 I’m hoping to see a much smoother i/o transfer without step-ups and without it being ‘choppy’, and in so doing the overall average transfer rates will be better/smoother and hopefully in the region of at least 800 to 1000 MBytes/sec continuously during the transfer when the data is moving from the server’s buffer cache to the client’s buffer cache.

Our MP6,1 server has fast RAID-5 than can deliver around 800 MBytes/sec. At times we place Project data in the server’s internal SSD (it’s a 1TB Apple PCIe SSD that delivers around 1400 MBytes/sec). The client MP6,1 systems only read the initial Project data files from the server and then do their work and eventually deliver/write back the results to the server in their respective locations. In this manner there never ever any chance Clients will be at odds with each other reading data from the server nor writing data to the server. Thus our MP6,1 File Server cannot be considered being a true File Server. So one might say we’ve configured a poor man’s server and fast data networking using the Thunderbolt Bridge technique.

I simply do not want to be at the mercy of Apple any longer as they simply seem to mess up the networking software layers now and then and at unexpected times for us doing Production work at the office.

I’m keeping my fingers crossed that these SANLink2 adapters work as well as everybody is saying here and elsewhere. They were not cheap at $535 each from my Apple Business Dept. If they don’t work I can return them within 14 days for a full refund.

I presume I need to use CAT-6 cable or better for connecting the two SANLink2 adapters, right ? I also assume I can bond two CAT6 cables for link aggregation and fail over, right ? Improves bandwidth but does not improve individual write and read speeds as that is done over a single cable.

Nice to hear from you, and thanks for responding.

[Chris Duffy]

Hey Barry,

ya, Thunderbolt Bridge is a %$% lol

and yes Apple keeps changing smb every release and
one never knows if it will work (haha) any better then afp
but sooner or later it should.

you can bond the two ports on a SANLink2 but keep in mind
as you mentioned that Mac OS will only use one of the ports
mostly and the other port while slightly used, i.e. on return
packets/acks and so forth, it does not buy you much.

The SANLink2’s will have a steady stream for performance
BUTTTTTT this only will hold true if the disk I/O of the client
and server do not get caught up in any bottlenecks, i.e.
if that Raid on the server is on a Thunderbolt (separate bus)
it helps but there is no guarantee….if you know what I mean.

Good luck,
Duffy

[Barry Sharp]

THANKS Chris for your responses.

Yes, bottle necks are always present and the trick is to unearth them and to figure out how to free them up…. and then of course you then bump into the next bottle neck…. and so on and so on. It’s simply the nature of the beast

In my case at the office what we do is to before starting up a Client’s work we preload its needed data into the servers kernel buffer cache (RAM). Now since RAM can deliver 5000+ MBytes/sec we position the server with its best delivery of data and make sure it’s not the bottle neck. Now the Client demands its data which is already in the servers RAM which then can be pushed across the network (today that’s the ethernet over Thunderbolt Bridge) to the Client’s kernel buffer cache (RAM). So in essence we get data moving from Server RAM to Client RAM across the TB Bridge. In this case the TB Bridge becomes the bottle neck as it can sustain no more than 20 Gbps (2.5 GBytes/sec)… but that’s pretty darn fast for our purposes. So this is all theory, right? What actually happens in our case today is that we have to choose carefully between SMB and AFP as that is what Apple provides with its file sharing feature. As you and I know, which you have alluded to, is that Apple mucks up either SMB or AFP or maybe both at same time. Apple’s file sharing software is what we have to deal with and over the years this has been a terrible chore. Today under both El Capitan and Sierra the SMB simply is not worth considering and I’m left with having to use AFP. The i/o across the TB Bridge takes a while to ‘ramp’ up when the server writes/pushes its data to the Client it does so in a very ‘choppy’ manner after the ‘ramp up’ completes. When the client writes data back to the server it goes a lot better but that is not of any real importance to me as that occurs only when the Client has finished its work and is delivering results back to the server. Other systems such as Linux and proprietary systems always seem to have a better handle on networking/ethernet performance compared to Apple’s efforts. IMO, Apple has little interest in networking as quite likely 99% of their customer base use a single Mac with at best a single external device to hold extra data. Thus we Pro folks that have the need for strong, fail-safe, reliable and fast networks are left to figure out how best to get our work done.

My hope is that the SANLink2 adapter will help ‘smooth’ the writes from server to client and offer an improved overall average i/o rate when the client systems ‘fire’ up.

My home test system uses a Late 2015 27″ iMac 5K Retina having 4.0 GHz i7 and 32GB RAM and 512GB PCIe internal SSD and a Promise Pegasus2 R6 RAID-0 (the server) and a Mid 2012 13″ MBP9,2 with 2.9 GHz i7, 16GB RAM and a fast internal SSD (the client). Both of these Macs are running Sierra 10.12.1. When I get the SANLink2 adapters I will first test things with the home system.

I’m trying to understand why the iMac can configure its Thunderbolt Bridge’s MTU to be 9000 while the MBP cannot, and is limited to 1500 for the MTU. This issue on the MBP is shown by the Network -> Advance… -> Hardware and by the Terminal command ‘networksetup’ as shown below…

bash-3.2# networksetup -listValidMTURange bridge0
Valid MTU Range: 1500-1500

Even by setting Configure: to “Automatically” the MTU box grays out and I don’t know if that will cause the MBP to use jumbo frame regardless.

At the office we have multiple Apple MP6,1s with 64GB RAM and 1TB SSDs inside. They all are running latest El Capitan OS. Last night I checked their Network panels for the Thunderbolt Bridge and they also are limited with an MTU = 1500 regardless of using Manual or Automatically.

I’m assuming MTU = 9000 is preferred over the 1500 value for best performance. Do you agree with this ?

I’m wondering if when connecting the SANLink2 adapters the MTU=1500 limitation will be lifted so I can use 9000.

Again, since both the Apple Thunderbolt Bridge and the Thunderbolt-to-SANLink2-to-10G are both converting Thunderbolt to Ethernet protocol my guess is that I will at best see less ‘choppy’ i/o and a smoother overall average i/o rate. That is, the SANLink2 will ‘smooth things out. What do you expect me to get going from Thunderbolt Bridge to using the SANLink2 ?

The one thing that bugs me is that Thunderbolt 2 is rated at 2.5 GBytes/sec whereas the SANLink2 using 10G only gives half of the Thunderbolt 2 rate. So given this will I observe better i/o rates using the SANLink2 ?

[Barry Sharp]

OK…. So I went to my local Apple Store where they have a MP6,1 running Sierra. Using the network setup in Terminal I was able to see that the MTU can be set to 9000. This tells me the issue related to MTU = 1500 limitation on our office MP6,1 systems is that they are running El Capitan. If we were to upgrade them to Sierra then MTU = 9000 will be possible.

Thus the MTU=1500 max limitation on the MP6,1 is related to the El Capitan OS and not any hardware limitation.

This is good new in some respects, but does mean I have to validate our workflow applications with Sierra which is just another chore on my plate…. sigh.

[Chris Duffy]

Barry,
all Mac OS levels and hardware will do 9000 MTU for the SANLINK2’s.
In the network preferences GUI,
select the interface/port,
change to
Configure->Manual
Speed-> autoselect

then the MTU can be set to 9000

duffy

[Barry Sharp]

Chris:

Is that true ONLY when the SANLink2 is connected. I ask this as my MBP9,2 which is running macOS Sierra does not allow MTU=9000 for the Thunderbolt Bridge with Manual whereas for 1GbE ethernet on the MBP9,2 the MTU=9000 can be set with Manual setting.

On my iMac17,1 running macOS Sierra Its Thunderbolt Bridge can be set for MTU=9000 with Manual as well as its 1GbE ethernet.