-
understanding limitations of CUBIX expansion?
I’m still trying to wrap my head around how using PCI expension chassis are possible.
(I’ll use a 2010 Mac Pro 8 core for this example…)
So my mac has 2 PCIe16x slots and two 8x slots. Therefore it has 40 total PCIe lanes of bandwidth. WHich I’m currently using all of.Now I can get an expansion chassis which somehow is magically adding 64 lanes to my motherboard, more than doubling its throughput. This makes no sense.
Or maybe I’m conceptualizing the problem wrong. Perhaps the lane allocation is purely a conceptual “allotment” unit for accessing system bandwidth. And slapping a couple nvidia n200 chipsets on a card can deal with splitting up the same system bandwidth across more lanes.
But if thats the case then the System’s QPI must be the bottleneck. So a 2010 8 core with the 5.6GT/s motherboard might work where a 2009 8 core with the 4.8GT/s motherboard may not. And as soon as a core piece of my system operates “mysteriously”, it becomes a black box that runs on voodoo. And I can’t fix voodoo when it breaks. Which it will.
I’d welcome any insight on this.
-
21 Replies
-
Hi, the basic concept is that the x16 slot that the expander uses shares the bandwidth to the four new slots…. if you assume that a GPU or other cards in those new slots require the full slot bandwidth for 100% of the cards duty cycle then you are correct that this ‘voodoo’ wont work. Fortunately the GPU and other cards need most of the frame duration to do their own internal magic and so the slot use by each card is manageable. That’s not to say there are no limitations, just too many variables to be able to check every single one. Thus the config guide describes known working combinations.
Peter -
Hi Blase,
Further to Peter’s response, I’d like to take you back in history to the days of the PowerMac G4 or older computers. These computers had a single PCI bus which was shared by multiple PCI cards. The Cubix expander is conceptually providing similar functionality by sharing a Mac Pro x16 lane PCI Express 2.0 slot between multiple PCI Express cards. The fast x16 lane PCI Express 2.0 slots in the Mac Pro make it easier than ever before to share multiple demanding cards on the one bus.
PCI Express cards regularly poll the PCI Express bus as they require it but they don’t require the bus all of the time. The PCI Express bus can be shared and other cards will use it when the first card isn’t using the bus. Obviously one shouldn’t attach too many cards to a single bus or else the cards will have to wait too long for each other to perform each job.
I hope this helps to demystify the black box voodoo.
Regards,
Luke Maslen
Blackmagic Design -
I’ll be honest, it still seems a bit vague.
You’re simply saying that the cards aren’t drawing a whole lot of bandwidth, so they can all coexist without saturating a single PCIe2x16 slot.
If this were a dynamics engine or something, that would make sense. The processing would be the biggest factor and the data bandwidth itself would be minimal. But we’re talking about 4k video processing. Its probably the most bandwidth intensive field there is.
Here’s a great explanation of my problem: Put a gtx285 in an x4 slot and try and run Resolve. It would run like crap. Conceptually that’s what you’re doing. Turning an x16 slot into 4 x4 slots with 4 graphics cards (or two x8 cards)?
If I’m missing something please help me see where.
Or rather if its too complicated to explain, please help me understand the consequences.
ie. –- Will a cubix box help add more realtime nodes for 2k but hinder 4k?
- Will adding a redrocket to a cubix box (that already has dual gtx285’s) be slower than adding it to the main board?
- Would there be diminishing results using 3 or 4 gtx285 cards as opposed to just 2?
Thanks,
Blase -
Funny that you used the word “voodoo.” Here’s my post from July when I was trying to find the cheapest PCI-e solution during the Color/Resolve hysteria (before we realized it wasn’t an issue.)
https://forums.creativecow.net/readpost/223/17874
Like I said months ago, audio guys have been using these for years with their ProTools HD systems. It’s not voodoo. I know it’s not video, but they were still running hundreds of tracks of audio YEARS ago with DSP cards in a chassis on G4’s.
Next one….
Borrowing from Rohit’s post on REDUSER…
“On a setup here, we are using 2 GTX285s and 1 Red Rocket in the Cubix expansion chassis, 4096×2304 Red Half-res premium decode, and more than 20 nodes of grades at 24fps. We also have a Decklink Extreme 3D+ and a ATI 5770 for GUI monitoring in the Mac Pro. There is still a slot open for a FC card in the Mac Pro.”
I think you should just email or call the guy at Cubix for a detailed explanation.
twitter.com/illyalaney
-
I called Cubix today..
I voiced my concern, that the x4 unit would essentially be splitting up a single x16 bus 4 ways. They explained that basically each device would have the full x16 bandwidth, but would only have access to it for 1/4th of the time.
So while the device itself does NOT in anyway create a bottleneck, the x16 slot on the motherboard does.Whether the bottleneck is significant or where returns becoming diminishing is yet to be tested.
Evidently they are shipping an x4 unit out to BM for testing right now, so the answers are coming.THE PROBLEM:
As I understand it, the layout suggested in the BM config guide is using an x4, but to my knowledge, BM has only tested this with dual x2 units.The issues I’m bringing up wouldn’t be a problem with the x2, but could be with the x4. And the layout I’d want could only be done with an x4 (or an x4 and x2).
it would be:
– GT120
– cubix ->
. GTX285
. GTX285
. Red rocket
. Red rocket (negotiable, but icing on cake)
– DL Extreme 3d
– RAID cardAnd ideally I’d want to know how it handles at 4k as well.
-
Why would you add a RAID card and DeckLink when you could throw 3 GTX 285’s in there? Just keep the RAID and DeckLink in the suggested slots on the MacPro.
I think all of these quotes explain everything you need to know about how this works.
From Luke
“PCI Express cards regularly poll the PCI Express bus as they require it but they don’t require the bus all of the time.”From you
“but would only have access to it for 1/4th of the time.”From an old post by Eric Fiegehen
“While the argument regarding bus traffic is certainly valid for some applications, Cubix has found that other applications from vendors such as Acceleware and Refractive software (Octane Render) execute mainly on the GPUs, not on the CPUs. Therefore, PCIe bus traffic is minimal and does not impact performance in most cases.twitter.com/illyalaney
-
[Illya Laney] “Why would you add a RAID card and DeckLink when you could throw 3 GTX 285’s in there? Just keep the RAID and DeckLink in the suggested slots on the MacPro.”
Because that has never actually been tested by anyone to work. Thats my point.
Also because I ABSOLUTELY need a redrocket card working in the system. And for any 4k work, I’d need 2.[Illya Laney] “From an old post by Eric Fiegehen
“While the argument regarding bus traffic is certainly valid for some applications, Cubix has found that other applications from vendors such as Acceleware and Refractive software (Octane Render) execute mainly on the GPUs, not on the CPUs. Therefore, PCIe bus traffic is minimal and does not impact performance in most cases.”Again you are not understanding the problem. In a render farm like referenced above, the CUBIX is ideal. Because processing is heavy and data bandwidth is minimal. We are the EXACT OPPOSITE of render farms, or computers simulations, or ProTools rigs. We are processing light and bandwidth HEAVY!
Cubix x2 seems to work fine for 2k. But assuming cubix x2 would not bottleneck on 4k, or that cubix x4 would not bottleneck in 2k or 4k, is still an untested assumption.
-
Blase
I can tell you from my experience that everything with the cubix is working perfectly. I have the Cubix in Slot 1 and GT120 in Slot 2 (per Rohit’s suggestion). Decklink in Slot 3 and my Fibre Card in Slot 4. In the Cubix I have 2 GTX 285s and believe it or not you can actually squeeze in a rocket (its very tight and I’m slightly worried about heat from the adjacent GTX) but it works even though the Config guide says it should be blocked.
Performance has been awesome. I got the setup mainly for stereo work that I’m now doing and the occasional RED job. I’ve been floored with the overall performance. I have not experienced any of the bottle necks you seem to be worried about in my config.
Robbie Carman
—————-
Colorist and Author
Check out my new Books:
Video Made on a Mac
Apple Pro Training Series DVDSP
From Still To Motion -
Cool this is great to hear!
My concern would be that you won’t hit a bottleneck until you actually load an r3D job, and the RR card starts to compete with the other 2 cards. Have you tested that yet?
-
[Blase Theodore] “Have you tested that yet?”
10 min grade heavy RED project starting Friday I’ll report back
Robbie Carman
—————-
Colorist and Author
Check out my new Books:
Video Made on a Mac
Apple Pro Training Series DVDSP
From Still To Motion
Reply to this Discussion! Login or Sign Up