[Barry Sharp]

Hi Bob:

Thanks for you reply and advice.

Question: Should the /etc/nsmb.conf have any contents or is it sufficient to just define/create it as a zero length file ?

1) Non optical TB cables can be had for $39 for a 6.6-foot cable, $48 for a 9.8-foot cable. Beyond that one has to go for the Corning optical cables that start at around $245 for a 33-foot cable.

2) If Thunderbolt 3 improves this for the ‘Bridge’ solution that will be very welcomed.

3) Yes, I have the SANLink2 Adapters now and started my testing yesterday using them. I did preload the Promise driver software which was done without issues.

4) Yes, as soon as connecting the SANLink2 adapters I was able to set MTU=9000 on my MBP. when configuring “Manual”.

5) I did discuss File Servers and 10G connectivity with Chris Duffy several years ago but at that time the cost of the minimum Small Tree file server would not fit our office budget at the time. The cost then was close to $10,000 including file server, switch, 10G adapters for the Macs (pre MP6,1 model) and cables. We limp along using 1GbE bonded pairs for our Mac for a while and then switch them all out for the Late 2013 MP6,1s.

6) All of the office MP6,1 and iMacs are situated at workstations in a circle around a enclosed central office. In the central office we have a maxed out MP6,1 12-core with as much RAM as we could ram into it along with a the largest internal SSD. This MP6,1 has all of our high-speed RAID-0, RAID-5 and backup units which are all connected to this MP6,1. We call it our File Server although strictly speaking it’s not in the technical sense. The cable distance between any Client Mac and the central inner office Mac is no more than 9 feet with a small amount of slack. In some cases the cable distance is 6 feet with some slack. Our Client Macs outside of the office pull/read from the inner office MP6,1 File Server their need project data, crunch on this data and eventually return results to the File Server Mac6,1. Each Client does this and grab their piece of the Project work and return their specific results. The File Server MP6,1 can then when all Clients have completed their workload and delivered results pull all the results together and produce final results.

7) The Clients use a Thunderbolt Bridge to the central MP6,1 File Server. Total cost was no more than $130.

8) The ‘Bridge’ works for us as the Clients ONLY read data from the File Server in the inner office for the most part to get ONLY their share of the Project work and deliver ONLY their result files. Because of this there’s no collision of data being written back to the File Server in the inner office.

9) Our solution is limited as there are only so many Thunderbolt ports on the MP6,1. But so far we can deal with having four Clients connected via Thunderbolt Bridge to the inner office File Server MP6,1. To go beyond we would need to obtain a ‘real’ file server such as SmallTree offers and use 10G cables, 10G switch and 10G adapters such as the SANLink2.

10) It should be noted that the 10G adapter all start off having to connect to the Mac via a Thunderbolt cable. For this reason, I expect the i/o patterns, dat flow, choppiness and stalling I see using the Thunderbolt Bridge to still be apparent.

11) Yesterday I ran my first set of tests using my home test rig of two Macs. I employed AJA for the testing as well as a few ‘dd’ execution for good measure. I used the old AJA version rather than the newest AJA version, but will also use the newer AJA over next few days to ensure the results are consistent (or not).

12) So here’s my test report

Summary
1) The Thunderbolt Bridge was far superior in both performance and cost vs. using the SANLink2 Adapters.
2) Let me say up front that using two SANLink2 Adapters for moving data between two Macs is likely an anathema to some people.
3) The SANLink2 Adapter in all fairness is a device for allowing a Mac to access a storage device (local or a NAS through a switch) that has 10GbE port(s).
4) My purpose for wanting to perform this evaluation was to see if the data transfers would be ‘smoother’ using the SANLink2 Adapters vs. the Thunderbolt Bridge which are seen to be ‘choppy’ at times. I really don’t expect the i/o rates to be better, but hopefully on a par with the Thunderbolt Bridge.
5) The storage device used was a Promise Pegasus2 R6 (RAID-0 across six 7200rpm 2TB Toshiba disks) connected directly via Thunderbolt-2.

In order to establish a baseline for the best performance and to prepare for reviewing how well a pair of PROMISE SANLink2 Thunderbolt 2 to 10 Gb/s Ethernet Adapters will compare to using a simple Thunderbolt Bridge between two Macs; a Mid 2012 13-inch MBP9,2 with Thunderbolt-1 and a Late 2015 27-inch iMac 5K Retina with Thunderbolt-2, I first ran AJA on the iMac to sweep File Sizes from 128MB to 16GB to the iMac’s connected Promise Pegasus2 R6. The R6 unit was 25% full.

Both Macs were running the latest macOS Sierra 10.12.1 (Build 16B2553a). The File Sharing was set for Apple’s AFP.

Topology: iMac <–> Thunderbolt Cable <–> Promise Pegasus2 R6

As a further sanity check I decided to quickly run the newest AJA….

Woweeee, this new AJA is very colorful and is showing much higher write/reads for my iMac using the Pegasus2 R6 directly.

Just tested writing and reading a 64GB file….. (not sure what all the settings are at this time)

At first blush using the new AJA System Test the results for Case 2) and 3) (Bridge vs. SANLink2) are close for writes and reads.

I’ll need more testing using the new AJA to ensure of my findings are correct. However, at this time the TB Bridge still is a better choice when you compare the Bridge cost of a mere $60 for a TB cable vs. a pair of SANLink2 Adapters costing $1,070 there really is no issue in deciding which to use.

Here’s Case 2) using TB Bridge

Here’s Case 3) using SANLink2

[Barry Sharp]

Hi Bob:

Thanks for you reply and advice.

1) Non optical TB cables can be had for $39 for a 6.6-foot cable, $48 for a 9.8-foot cable. Beyond that one has to go for the Corning optical cables that start at around $245 for a 33-foot cable.

2) If Thunderbolt 3 improves this for the ‘Bridge’ solution that will be very welcomed.

3) Yes, I have the SANLink2 Adapters now and started my testing yesterday using them. I did preload the Promise driver software which was done without issues.

4) Yes, as soon as connecting the SANLink2 adapters I was able to set MTU=9000 on my MBP. when configuring “Manual”.

5) I did discuss File Servers and 10G connectivity with Chris Duffy several years ago but at that time the cost of the minimum Small Tree file server would not fit our office budget at the time. The cost then was close to $10,000 including file server, switch, 10G adapters for the Macs (pre MP6,1 model) and cables. We limp along using 1GbE bonded pairs for our Mac for a while and then switch them all out for the Late 2013 MP6,1s.

6) All of the office MP6,1 and iMacs are situated at workstations in a circle around a enclosed central office. In the central office we have a maxed out MP6,1 12-core with as much RAM as we could ram into it along with a the largest internal SSD. This MP6,1 has all of our high-speed RAID-0, RAID-5 and backup units which are all connected to this MP6,1. We call it our File Server although strictly speaking it’s not in the technical sense. The cable distance between any Client Mac and the central inner office Mac is no more than 9 feet with a small amount of slack. In some cases the cable distance is 6 feet with some slack. Our Client Macs outside of the office pull/read from the inner office MP6,1 File Server their need project data, crunch on this data and eventually return results to the File Server Mac6,1. Each Client does this and grab their piece of the Project work and return their specific results. The File Server MP6,1 can then when all Clients have completed their workload and delivered results pull all the results together and produce final results.

7) The Clients use a Thunderbolt Bridge to the central MP6,1 File Server. Total cost was no more than $130.

8) The ‘Bridge’ works for us as the Clients ONLY read data from the File Server in the inner office for the most part to get ONLY their share of the Project work and deliver ONLY their result files. Because of this there’s no collision of data being written back to the File Server in the inner office.

9) Our solution is limited as there are only so many Thunderbolt ports on the MP6,1. But so far we can deal with having four Clients connected via Thunderbolt Bridge to the inner office File Server MP6,1. To go beyond we would need to obtain a ‘real’ file server such as SmallTree offers and use 10G cables, 10G switch and 10G adapters such as the SANLink2.

10) It should be noted that the 10G adapter all start off having to connect to the Mac via a Thunderbolt cable. For this reason, I expect the i/o patterns, dat flow, choppiness and stalling I see using the Thunderbolt Bridge to still be apparent.

11) Yesterday I ran my first set of tests using my home test rig of two Macs. I employed AJA for the testing as well as a few ‘dd’ execution for good measure. I used the old AJA version rather than the newest AJA version, but will also use the newer AJA over next few days to ensure the results are consistent (or not).

12) So here’s my test report

Summary
1) The Thunderbolt Bridge was far superior in both performance and cost vs. using the SANLink2 Adapters.
2) Let me say up front that using two SANLink2 Adapters for moving data between two Macs is likely an anathema to some people.
3) The SANLink2 Adapter in all fairness is a device for allowing a Mac to access a storage device (local or a NAS through a switch) that has 10GbE port(s).
4) My purpose for wanting to perform this evaluation was to see if the data transfers would be ‘smoother’ using the SANLink2 Adapters vs. the Thunderbolt Bridge which are seen to be ‘choppy’ at times. I really don’t expect the i/o rates to be better, but hopefully on a par with the Thunderbolt Bridge.
5) The storage device used was a Promise Pegasus2 R6 (RAID-0 across six 7200rpm 2TB Toshiba disks) connected directly via Thunderbolt-2.

In order to establish a baseline for the best performance and to prepare for reviewing how well a pair of PROMISE SANLink2 Thunderbolt 2 to 10 Gb/s Ethernet Adapters will compare to using a simple Thunderbolt Bridge between two Macs; a Mid 2012 13-inch MBP9,2 with Thunderbolt-1 and a Late 2015 27-inch iMac 5K Retina with Thunderbolt-2, I first ran AJA on the iMac to sweep File Sizes from 128MB to 16GB to the iMac’s connected Promise Pegasus2 R6. The R6 unit was 25% full.

Both Macs were running the latest macOS Sierra 10.12.1 (Build 16B2553a). The File Sharing was set for Apple’s AFP.

Topology: iMac <–> Thunderbolt Cable <–> Promise Pegasus2 R6

As a further sanity check I decided to quickly run the newest AJA….

Woweeee, this new AJA is very colorful and is showing much higher write/reads for my iMac using the Pegasus2 R6 directly.

Just tested writing and reading a 64GB file….. (not sure what all the settings are at this time)

At first blush using the new AJA System Test the results for Case 2) and 3) (Bridge vs. SANLink2) are close for writes and reads.

I’ll need more testing using the new AJA to ensure of my findings are correct. However, at this time the TB Bridge still is a better choice when you compare the Bridge cost of a mere $60 for a TB cable vs. a pair of SANLink2 Adapters costing $1,070 there really is no issue in deciding which to use.

Here’s Case 2) using TB Bridge

Here’s Case 3) using SANLink2

[Barry Sharp]

I will have my two SANLink2 adapters later today so will start testing then. Will report back with results.

[Barry Sharp]

Chris:

Is that true ONLY when the SANLink2 is connected. I ask this as my MBP9,2 which is running macOS Sierra does not allow MTU=9000 for the Thunderbolt Bridge with Manual whereas for 1GbE ethernet on the MBP9,2 the MTU=9000 can be set with Manual setting.

On my iMac17,1 running macOS Sierra Its Thunderbolt Bridge can be set for MTU=9000 with Manual as well as its 1GbE ethernet.

[Barry Sharp]

OK…. So I went to my local Apple Store where they have a MP6,1 running Sierra. Using the network setup in Terminal I was able to see that the MTU can be set to 9000. This tells me the issue related to MTU = 1500 limitation on our office MP6,1 systems is that they are running El Capitan. If we were to upgrade them to Sierra then MTU = 9000 will be possible.

Thus the MTU=1500 max limitation on the MP6,1 is related to the El Capitan OS and not any hardware limitation.

This is good new in some respects, but does mean I have to validate our workflow applications with Sierra which is just another chore on my plate…. sigh.

[Barry Sharp]

THANKS Chris for your responses.

Yes, bottle necks are always present and the trick is to unearth them and to figure out how to free them up…. and then of course you then bump into the next bottle neck…. and so on and so on. It’s simply the nature of the beast

In my case at the office what we do is to before starting up a Client’s work we preload its needed data into the servers kernel buffer cache (RAM). Now since RAM can deliver 5000+ MBytes/sec we position the server with its best delivery of data and make sure it’s not the bottle neck. Now the Client demands its data which is already in the servers RAM which then can be pushed across the network (today that’s the ethernet over Thunderbolt Bridge) to the Client’s kernel buffer cache (RAM). So in essence we get data moving from Server RAM to Client RAM across the TB Bridge. In this case the TB Bridge becomes the bottle neck as it can sustain no more than 20 Gbps (2.5 GBytes/sec)… but that’s pretty darn fast for our purposes. So this is all theory, right? What actually happens in our case today is that we have to choose carefully between SMB and AFP as that is what Apple provides with its file sharing feature. As you and I know, which you have alluded to, is that Apple mucks up either SMB or AFP or maybe both at same time. Apple’s file sharing software is what we have to deal with and over the years this has been a terrible chore. Today under both El Capitan and Sierra the SMB simply is not worth considering and I’m left with having to use AFP. The i/o across the TB Bridge takes a while to ‘ramp’ up when the server writes/pushes its data to the Client it does so in a very ‘choppy’ manner after the ‘ramp up’ completes. When the client writes data back to the server it goes a lot better but that is not of any real importance to me as that occurs only when the Client has finished its work and is delivering results back to the server. Other systems such as Linux and proprietary systems always seem to have a better handle on networking/ethernet performance compared to Apple’s efforts. IMO, Apple has little interest in networking as quite likely 99% of their customer base use a single Mac with at best a single external device to hold extra data. Thus we Pro folks that have the need for strong, fail-safe, reliable and fast networks are left to figure out how best to get our work done.

My hope is that the SANLink2 adapter will help ‘smooth’ the writes from server to client and offer an improved overall average i/o rate when the client systems ‘fire’ up.

My home test system uses a Late 2015 27″ iMac 5K Retina having 4.0 GHz i7 and 32GB RAM and 512GB PCIe internal SSD and a Promise Pegasus2 R6 RAID-0 (the server) and a Mid 2012 13″ MBP9,2 with 2.9 GHz i7, 16GB RAM and a fast internal SSD (the client). Both of these Macs are running Sierra 10.12.1. When I get the SANLink2 adapters I will first test things with the home system.

I’m trying to understand why the iMac can configure its Thunderbolt Bridge’s MTU to be 9000 while the MBP cannot, and is limited to 1500 for the MTU. This issue on the MBP is shown by the Network -> Advance… -> Hardware and by the Terminal command ‘networksetup’ as shown below…

bash-3.2# networksetup -listValidMTURange bridge0
Valid MTU Range: 1500-1500

Even by setting Configure: to “Automatically” the MTU box grays out and I don’t know if that will cause the MBP to use jumbo frame regardless.

At the office we have multiple Apple MP6,1s with 64GB RAM and 1TB SSDs inside. They all are running latest El Capitan OS. Last night I checked their Network panels for the Thunderbolt Bridge and they also are limited with an MTU = 1500 regardless of using Manual or Automatically.

I’m assuming MTU = 9000 is preferred over the 1500 value for best performance. Do you agree with this ?

I’m wondering if when connecting the SANLink2 adapters the MTU=1500 limitation will be lifted so I can use 9000.

Again, since both the Apple Thunderbolt Bridge and the Thunderbolt-to-SANLink2-to-10G are both converting Thunderbolt to Ethernet protocol my guess is that I will at best see less ‘choppy’ i/o and a smoother overall average i/o rate. That is, the SANLink2 will ‘smooth things out. What do you expect me to get going from Thunderbolt Bridge to using the SANLink2 ?

The one thing that bugs me is that Thunderbolt 2 is rated at 2.5 GBytes/sec whereas the SANLink2 using 10G only gives half of the Thunderbolt 2 rate. So given this will I observe better i/o rates using the SANLink2 ?

[Barry Sharp]

Chris:

Yes, sorry for not responding to you saying…”well, well etc”. I’m now retired from HPC and do all the IT at local office for my son and my consulting work. We are an all Mac setup with MP6,1 12core and beastie iMacs.

We’ve been getting along kind of AOK using a simple Thunderbolt Bridge between two of our MP6,1 systems and for the most part it works except that Apple keeps tuning/tweaking/braking the networking software layers (SMB and/or AFP) moving from one OS version to the next. Very annoying. We had been using SMB for the TB Bridge prior to El Capitan and Sierra. We now find that AFP provides a much much better i/o rate. It means with each macOS version/update we have to test the TB Bridge i/o to make sure there’s been no regression. The TB Bridge certainly is a low cost solution and can move data at well over 1000 MBytes/sec in spurts, but cannot maintain that over an entire file transfer. It’s choppy and this results in the overall average i/o rate to be a lot lower. This 1000 MBytes/sec is when the MP6,1 server system (which has 64GB RAM) has most, if not all, of the data in its kernel buffer cache which can deliver well over 5000 MBytes/sec. This 5000 MBytes/sec obviously overloads the Thunderbolt-2 by a factor of 2, but does show as much as 1500 MBytes/sec at times.

With the use of the SANLink2 I’m hoping to see a much smoother i/o transfer without step-ups and without it being ‘choppy’, and in so doing the overall average transfer rates will be better/smoother and hopefully in the region of at least 800 to 1000 MBytes/sec continuously during the transfer when the data is moving from the server’s buffer cache to the client’s buffer cache.

Our MP6,1 server has fast RAID-5 than can deliver around 800 MBytes/sec. At times we place Project data in the server’s internal SSD (it’s a 1TB Apple PCIe SSD that delivers around 1400 MBytes/sec). The client MP6,1 systems only read the initial Project data files from the server and then do their work and eventually deliver/write back the results to the server in their respective locations. In this manner there never ever any chance Clients will be at odds with each other reading data from the server nor writing data to the server. Thus our MP6,1 File Server cannot be considered being a true File Server. So one might say we’ve configured a poor man’s server and fast data networking using the Thunderbolt Bridge technique.

I simply do not want to be at the mercy of Apple any longer as they simply seem to mess up the networking software layers now and then and at unexpected times for us doing Production work at the office.

I’m keeping my fingers crossed that these SANLink2 adapters work as well as everybody is saying here and elsewhere. They were not cheap at $535 each from my Apple Business Dept. If they don’t work I can return them within 14 days for a full refund.

I presume I need to use CAT-6 cable or better for connecting the two SANLink2 adapters, right ? I also assume I can bond two CAT6 cables for link aggregation and fail over, right ? Improves bandwidth but does not improve individual write and read speeds as that is done over a single cable.

Nice to hear from you, and thanks for responding.

[Barry Sharp]

So my attached graph did not make it into my posting!

Thanks for the reply and the advice about the need to download software driver code from Promise site. I’ll do that ahead of time.

I’ll try attaching my graph again here.

This graph is me using a Thunderbolt Bridge between my home MBP and my home iMac.

The i/o traffic writing and reading across my Thunderbolt Bridge using my MBP as Server and my iMac as Client. I performed the i/o from the MBP. This was using AFP which performs better than does SMB under Sierra 10.12.1 (build 16B2584a). Even so, you can see how the writes run fairly smoothly but ramp up over the course of writing in two distinct steps (the orange graph). For reading from the iMac (the blues graph) you can see how choppy the transfer is, and also note how it ramps up initially before its gets going.

The peak write (orange) achieved here was around 800 MBytes/sec and the peak read (blue) tops out at around 1350 MBytes/sec. However the average rate is much much lower because of all the writing step-ups and the stops & starts in the reading transfer.

So I’m hoping/expecting the SANLink2 will avoid these writing steps and the ‘choppy’ reads. Should this expectation be real I ask ?

BTW…. on my MBP I cannot set MTU to 9000 no matter what I do. On the iMac I can set MTU to 9000. Why is this ?

Can I use a command in Terminal on the MBP to force/setup MTU=9000 ? If so, what would this commend be ?

[Barry Sharp]

I have just ordered two PROMISE SANLink2 Thunderbolt 2 to 10 Gb/s Ethernet Adapters. My intent is to use these for two of my office Apple MP6,1 systems.

I will also use them for testing at first using my home Apple Late 2015 iMac17,1 27″ 5K Retina (32GB RAM) and my Mid 2012 Apple MBP9,2. (16GB RAM) The iMac has Thunderbolt 2 and the MBP has Thunderbolt 1.

The office MP6,1 systems have Thunderbolt 2 and 64GB RAM.

All the Macs are running macOS Sierra.

I notice on the MBP’s Network panel “Advanced…” I cannot set MTU above 1500, whereas on the iMac I can set MTU to 9000. I’ve not checked this on the MP6,1 system at this time.

The reason for trying out these SANLink2 devices is to see if the data transfer between our two office MP6,1 systems is better than using a simple Thunderbolt Bridge. This technique is very cost effective as it requires just a single Thunderbolt cable. However, we see varying issues over time using the Apple’s SMB and AFP. Sometimes SMB is better and sometimes AFP is better. It seems to vary with the macOS version we run. Currently running Sierra we find AFP is much better than SMB whereas pre Sierra we found SMB was better. The other issue we see with using the tHunderbolt Bridge is that the i/o rate is VERY choppy, and at this time is very bad using SMB vs. using AFP. I attach a graph (hopefully I’ve done this correctly) showing what I’m seeing using AFP. The orange is for writing from my MPB to the iMac and the blue is for reading from the iMac to the MBP.

So the writing goes along and improves in two distinct steps which I don’t understand (that is, why the sudden steps). The reading shows an initial ramp up and then continues to be very ‘choppy’ with lots of stalls.

I’m hoping the SANLink2 will address these issues and smooth out the data transfers and consequently provide a better overall average i/o rate.

Question: Does the SANLink2 require special driver code that needs to be installed, or is it a plug&play setup much like for the Sonnet Adapter which I’m told does not require driver code to be installed ?

Thanks for any attention to this issue(s).