understanding limitations of CUBIX expansion?

[Nick Hasson]

I have my cubix desktop 4 expander coming next week. I have to figure out how to use the slots.

I really want to get my 10g ethernet adapter into a slot faster than 4x. The 4x slot is killing performance.

These are the cards I will have.

2 -FX4800 cards
gt120
decklink 3d
Small tree 10gig card
raid HBA

I’m thinking the mac will have
slot 1 – expansion
slot 2 – gt120
slot 3 – Decklink
slot 4 – Raid HBA

Then in the cubix
slot 1 – FX4800
Slot 2 – FX4800
slot 3 – small tree 10g
slot 4 – open for now.

Nick Hasson
Smoke/RESOLVE
http://www.niceedits.com

[Illya Laney]

Blase Theodore
“We are the EXACT OPPOSITE of render farms, or computers simulations, or ProTools rigs. We are processing light and bandwidth HEAVY!”

Accelware and Octane Render are not render farms.

Maybe you’re getting confused because you think video applications like FCP work the same as Resolve or Color. Most of DaVinci’s load is on the GPU’s and like Eric Fiegehen from Cubix said “PCIe bus traffic is minimal.” If you were right about all this, then why does it work? I’ve personally seen multiple cards used in a Cubix months ago(before multi GPU support) and it worked fine.

BTW, DSP’s for ProTools rigs are basically the video equivalent of a dedicated GPU for processing so in fact it is not the opposite of a ProTools system.

twitter.com/illyalaney

[Blase Theodore]

[Illya Laney] “Accelware and Octane Render are not render farms.”

Yes, they are. They are computational and physics-based render engines respecitvely.
https://www.refractivesoftware.com/

[Illya Laney] “Most of DaVinci’s load is on the GPU’s and like Eric Fiegehen from Cubix said “PCIe bus traffic is minimal.””

Eric was referring to Accelware and Octane Render in that statement, which again is the opposite of what we’re doing.

• With a render engine like Octane, the CPU is passing a few lines of code through the PCIe bus to the graphics card, which in turn does massive calcuations on a small volume of data.
• A ProTools rig takes a tiny audio sample, probably a few Kb, and runs it through powerful DSP’s.
• Resolve on the other hand has to deal with an uncompressed 4k image buffer cycling 24 times a second.

This is not an “apples for apples” comparison.

I’m not saying it won’t work, but I am saying its a legitimate question, and glazing over it does no one any good.

Robbie’s going to test it next week, which is as definitive an answer as I could ask for.
It will answer both the 4k and R3d concern at the same time.

[Illya Laney]

You don’t understand what qualifies as a Render Farm. A real time rendering engine(Octane and Accelware) is not the same as a render farm and they’re made for completely different purposes. So no, they’re not render farms. Here’s the link to the whole thread that Eric posted in, it should make things clear.

https://forums.creativecow.net/thread/277/260

Blase Theodore
“A ProTools rig takes a tiny audio sample, probably a few Kb, and runs it through powerful DSP’s.
Resolve on the other hand has to deal with an uncompressed 4k image buffer cycling 24 times a second.”

There are 48, 96, and 192 thousand samples a second going on for post audio. It wouldn’t be a few Kb a second if you’re running 24 bit 48Khz audio with 192 tracks in realtime. Despite the size, the concept is the same whether we’re talking about 4K or hundreds of tracks of audio playing. It’s all media being processed for real time performance so it’s not the “EXACT OPPOSITE” as you stated previously. When did this discussion become about 4K anyway? We were originally talking about a basic Cubix setup and how you thought it was voodoo.

Did you read my quote from Rohit earlier? Here’s what you should expect performance wise.

“”On a setup here, we are using 2 GTX285s and 1 Red Rocket in the Cubix expansion chassis, 4096×2304 Red Half-res premium decode, and more than 20 nodes of grades at 24fps. We also have a Decklink Extreme 3D+ and a ATI 5770 for GUI monitoring in the Mac Pro. There is still a slot open for a FC card in the Mac Pro.””

twitter.com/illyalaney

[Illya Laney]

And one more thing I should have mentioned…

https://en.wikipedia.org/wiki/Nvidia_Quadro_Plex

twitter.com/illyalaney

[Margus Voll]

I would not worry saturating one pcie.
In theory if 1 and 2 slots both use separate “channel” for data one could even use second
expander.

It is like with hard drives on ata ports used to be. Put one drive on a channel
and you get faster results. Here expander is like a drive. You put second one on second
channel and there you go.

Number of cards should not matter so much as duty cycle is so incredibly fast
that every card could do their job and move data and have some spare time left.

We just do not realize how fast that happens. 4k frame is not also very big compared to
data rate bus has. I may be wrong here but i remember something 50 megs per frame on rgb 4k.

—

Margus

https://iconstudios.eu

[Blase Theodore]

I think you’re probably right, but I hate hardware mysteries.

Here are some numbers..

The 2k data is probably around 3Gb/s (think dual link HD-SDI). And stereo 2k should be about 6Gb/s. But a 4k stream should be around 4x that. Lets say 12Gb/s. And then stereo 4k would be 24Gb/s.

A single PCI 2.x16 slot should provide 16Gb/s upstream and 16Gb/s downstream. So the data flow might be..

2k RED:
system -> 0.1Gb/s -> Redrocket
RR decode – > 3Gb/s -> system
system – > 3gb/s -> Dual GPU’s
GPU’s -> 3Gb/s -> system

Total upsteam/s: 3 of 16Gb
Total downstream/s: 6 of 16Gb
fine

Stereo 2k RED:
system -> 0.1Gb/s -> Redrocket
RR decode – > 6Gb/s -> system
system – > 6gb/s -> Dual GPU’s
GPU’s -> 6Gb/s -> system

Total upsteam/s: 6 of 16Gb
Total downstream/s: 12 of 16Gb
fine

4k RED:in 2k timeline
system -> 0.1Gb/s -> Redrocket
RR decode – > 12Gb/s -> system
system – > 3gb/s -> Dual GPU’s
GPU’s -> 3Gb/s -> system

Total upsteam/s: 12 of 16Gb
Total downstream/s: 6 of 16Gb
fine

4k RED:
system -> 0.1Gb/s -> Redrocket
RR decode – > 12Gb/s -> system
system – > 12gb/s -> Dual GPU’s
GPU’s -> 12Gb/s -> system

Total upsteam/s: 12 of 16Gb
Total downstream/s: 24 of 16Gb
problem

stereo 4k RED:
system -> 0.1Gb/s -> Redrocket
RR decode – > 24Gb/s -> system
system – > 24gb/s -> Dual GPU’s
GPU’s -> 24Gb/s -> system

Total upsteam/s: 24 of 16Gb
Total downstream/s: 48 of 16Gb
problem

Again, I recognize that this is already being successfully used, but everyone seems to be happy to glaze over the details, which will be important as soon as we get into 4k. At which point I might regret not having bought dual cubix instead of one x4.

[Margus Voll]

My idea was to use 2 cubix boxses on slot 1 and 2.
But ok it does not help if one slot gets saturated.

—

Margus

https://iconstudios.eu

[Blase Theodore]

[Margus Voll] “My idea was to use 2 cubix boxses on slot 1 and 2.”

Right exactly. That may be the appropriate solution (if there was indeed ever a problem.)

[Anders Rydmell]

Hi!

I think you have mixed Gb/s (gigabits/sec) with GB/s (gigabytes/sec) for the PCI Express bandwidth.
One PCI Express Gen2 x16 lane slot has 8 GB/s of bandwidth. That is equal to 64 Gb/s.

See https://en.wikipedia.org/wiki/Data_rate_units and https://en.wikipedia.org/wiki/Pci_express#PCI_Express_2.0 and https://www.intel.com/Assets/PDF/datasheet/321328.pdf section 1.1.4

The Mac Pro 2010 is using the Intel 5520 chipset with a total of 36 PCI Express Gen2 lanes.