-
SFP+ Cable Issues, Qbiquit and Mellanox
I’m having issues with performance between an iMac Pro and a QNAP NAS.
QNAP checked my configurations, found no issues.
QNAP ran some performance checks interally and determined the HW setup is fast.
QNAP ran some low level network test and confimed the slow performance I’m getting.
QNAP elevated the issue.I then talked to Ubiquiti, where they asked about the SFP+ cable I used. When I checked, I noticed that while it said it would work with Ubiquiti products, but no Mellanox. They did have another cable that would supposedly work with both.
QNAP is using Mellanox. Though to me, not work means nothing should work. My set up works, just slow.
So, just wondered if anyone here came across these kind of cabling issues.
-
26 Replies
-
Sometimes it can be a simple MTU settings. I typically do not use 1500 and use Jumbo frames 9000 MTU.
I’d run iperf and post the results.
-
Yep, check MTU – the mellanox driver defaults to MTU 9000 in some environments, seen that before.
Ifconfig or ethtool on the linux side if poss, see what it’s saying 🙂
-
EDIT: I woke up and realised I was thinking about the Myricom driver – this *does* set the interface to jumbo frames by default. Mellanox probably won’t.
Either way check you don’t have a mismatch, it’ll cause headaches.
-
Both the iMac and the QNAP server needed to be changed to Jumbo frames, 9000 MTU. However, makeing the changes had no effect on performance.
-
Hi Kevin,
Well that was unfortunate. Typically MTU settings are the issue. Is there a switch involved? If so send me the make and model and remember that MTU settings do need to be made on the switch.
Also I’d run iperf on the both directions server to client and then client to server. Iperf version 2 seems to be best and also supported under Mellanox so if you open a case with them they’ll want to see the version results.
The point of this test is to see if there is performance lost on both send and receive as some systems are setup to recieve and others to send.
Iperf works most OS’s too. Pretty simple to use
Server: iperf -s
Client: iperf -c server_ip -P 4The capital P means how many processes to run, in this case we’ll run 4 I’d also do 3 and just the default. Lowercase p would to specify the port if the default is different.
-
Steve,
On a long call with QNAP, their tech guy did run iPerf. (something I’m not familiar with)
The iPerf results were basically the same as all the other basic tests. (Blackmagic and AJA)
Though I’m not sure if he ran it from the server to the iMac Pro. He’s esculating it to QNAP HQ. Any advice on how to run iPerf on the server side would be appreciated.
For reference:
~ 560 MB/sec Write (many test sources confirm)
~ 350 MB/sec Read (many test sources confirm)
~ 630 MB/sec Read (testing the HW only within the server, so this is basically max perf)It’s odd that the read speeds are less (a lot too) than the write, as QNAP marketing posts the opposite.
I realize their is always overhead that keeps any real world results from getting anywhere near what the marketing guy says is theoretical maximum. But the read speeds are about half what they should be. Sadly read speed are rather important when editing.
-
Hmmm. Well the writes being faster actually doesn’t surprise me too much. I think qnap uses ZFS which is a copy on write file system. Means it uses ram and or fast SSDs to cache the info then it rights it to the actual disk.
Do you have anything else plugged in to the iMac that uses TB? If so try testing without any other TB device connected.
Do you have another system with TB that you could also test with?
-
wow –
lots of opinions, without asking many questions.# 1 – exactly what QNAP model do you have ?
# 2 – QNAP checked your configuration ? There are a million configurations. Are you in a single static volume with all 8, 12 or 16 drives, or do you have a storage pool. And if you have a pool, is it thick volume or thin volume ?
If you have a 2 bay, 4 bay or 6 bay QNAP – forget it – you are wasting your time. I don’t care what the QNAP salespeople or your dealer told you. You MUST have a minimum of EIGHT 7200 RPM drives spinning in a single RAID group to make this work with speeds fast enough for video editing.#3 – how are you connecting to the iMac Pro ? The iMac Pro has a built in 10Gbase-T port. You should have a
10GbaseT card for your QNAP. All modern QNAP’s have the ability to accept a PCIe card. The QNAP QXG-10G1T is only $89. It’s the exact same Aquantia AQC-107 chip that is in your iMac Pro. Perfectly compatible.#4 – you said SFP+ cable issues ! So with your iMac Pro that has a free 10G port, you purchased a thunderbolt 3 to SFP+ adaptor. Well that is wrong. Now, for the moment, I am going to ASSUME that you do not have one of the larger industrial QNAP products like the TVS-EC1680U-SAS-RP, which is a 16 bay with a dual port SFP+ card that comes standard. I am going to ASSUME that you have purchased a cheap QNAP TS-832X, or any Annapurna ARM based CPU. These QNAP products are GARBAGE and no matter what you do – no matter what QNAP told you – you are NEVER EVER EVER going to get good performance from them. If you don’t have an Intel or AMD based QNAP with EIGHT drives, you are going to fail.
You could have simply asked me before buying, and purchase a TVS-873e for about $1250, and a QNAP
QXG-10G1T, plugged this into your iMac Pro, and you would be getting 1000 MB/sec READ (500 MB/sec write).Now, let me assume again that you purchase a QNAP Product that their own press says “is great for video editing” –
like the TVS-453BT3 – well, they are lying. This product is NOT suitable for professional video editing.
QNAP makes TONS of products that are wonderful (and so does Synology, and so does Netgear) – and at the same time, they make “budget” products that are ok for running Plex server, and your home videos from your iPhone, but are NOT SUITABLE for professional video editing.AND ON TOP OF ALL OF THIS – if you did not disable SMB signing on your iMac Pro and you are connecting via SMB, you are going to get crappy speeds.
Well – until you tell me the EXACT model that you purchased, and the exact model of SFP+ adaptor you have (and I sure hope its not an Akitio) – then I can give you further analysis of what your problems are. But like I said – if you have a 2 bay, 4 bay or 6 bay product – or if you have an Annapurna based QNAP product – it’s never going to do what you want it to do.
You buy the right product and it will be great. And QNAP makes some great stuff (and they make some not so great stuff).
OH – and take that Ubiquiti switch out of the equation until you get this resolved. It’s just adding more to the confusion of diagnosing all of this.
Bob Zelin
I’m having issues with performance between an iMac Pro and a QNAP NAS.
QNAP checked my configurations, found no issues.
QNAP ran some performance checks interally and determined the HW setup is fast.
QNAP ran some low level network test and confimed the slow performance I’m getting.
QNAP elevated the issue.I then talked to Ubiquiti, where they asked about the SFP+ cable I used. When I checked, I noticed that while it said it would work with Ubiquiti products, but no Mellanox. They did have another cable that would supposedly work with both.
QNAP is using Mellanox. Though to me, not work means nothing should work. My set up works, just slow.
Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com -
Hi Bob. Thanks for replying.
Given the volume of help you provide here on the Cow, you probably don’t recall helping me out a few weeks ago when I first set up our QNAP. (see my configuration below) I started this post to specifically discuss the SFP+ cable.
Interesting that the QXG-10G1T card you recomend has the same controller as my iMac Pro. I’d gladly replace the pre-installed card if this either addressed the performance
issue, or simply made the system happier.While we probably can’t rule out the Ubiquiti switch as being an issue, it does allow write speeds much higher than read speeds. Could a switch even tell the difference?
QNAP checked to make sure that I had the correct MTU/Jumbo frames set correctly on both the iMac Pro and the server, which I did.
At this point, QNAP and Ubiquiti are esculating this performance issue. I hope to hear back from QNAP Monday.
I did order another SFP+ cable. This new version (from the same vendor) does state that it’s compatible with both Ubiquiti and the Mellanox controller. I don’t have a high level of confidence this will address the issue. I’d guess this is a works or doesn’t work issue, as opposed to a write is faster than read issue.
TS-873U-RP
Dual Port 10 Gbe SFP+ (preinstalled)
– I’m guessing it’s the LAN-10G2SF-MLX
– Mellanox MT27520
8 x Hitachi 8 TB Drives, HUH728080ALE600
One Static Volume
(no Storage Pool)10Gtek Cable from QNAP to Switch
This is the cable in question, as 10Gtek says this is compatible with Ubiquiti but not compatible with the Mellanox chip in the QNAP 10Gbe card.US‑16‑XG UniFi 10Gbe Switch
iMac Pro connected via RJ-45 Cat 6 to Switch
SMB:
I followed Apple’s published article about disabling SMB signing. It appears this is off by default on iMac Pro models, since I didn’t have to change anything.Per Apple: (I made no change to this file, it was already set to no)
Open the /etc/nsmb.conf file.
Set the signing_required value to “no,” like this:
[default]
signing_required=no
Save the /etc/nsmb.conf file.
Disconnect and then reconnect any mounted SMB shares to make the changes take effect.MTU Jumbo Frame:
iMac Pro (sys prefs)
– Configure Manually
– 10GbaseT
– full-duplex
– Jumbo (9000)
– (checked) AVB/EAV Mode
QNAP
– Jumbo Frame 9000 -
you are going to order the QNAP QXG-10G1T right now. End of story. You will spend the $89
plug the iMac Pro directly into the QXG-10G1T. Assign static IP address for the QNAP like
192.168.2.3
assign the iMac Pro a static IP address like 192.168.2.11
Jumbo Frames MTU9000 on both.Now do AJA Speed Test or Blackmagic Disk Speed Test. Now what do you get ?
Bob Zelin
ps – I use 10GTek Twinax cables all the time. They work great.
Bob Zelin
Rescue 1, Inc.
bobzelin@icloud.com
Reply to this Discussion! Login or Sign Up