Compressor Clusters

[Walter Soyka]

[Franco Bogino] “Can anyone shed some light on the benefits of using Virtual Clusters with Compressor & Qmaster? I set up a virtual cluster on a Mac Pro ‘Nehalem’ Quad core, instructing Compressor to use 4 instances. I tested it out on a 3min HVD clip, transcoding to ProRes. The cluster setup took 3 min 45 sec, while the normal render took 1 min 57 sec. “

Virtual clusters are supposed to speed up your encodes, but it sounds like you are pushing your system too hard. If you set it to use four instances on a quad-core machine, you risk overscheduling your processors. Try turning it down to 2 or 3 instances and see if that helps. For example, I use 6 instances on my 8-core machine.

RAM starvation is another possibility; how much RAM do you have?

[Franco Bogino] “Is it something to do with ‘stitching time’, and does this mean that clusters only save time when dealing with footage over a certain length?”

Stitching does add time to the encode, but hopefully that will be offset by the decrease in processing time.

Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog – What I’m thinking when my workstation’s thinking
Creative Cow Forum Host: Live & Stage Events

[Franco Bogino]

Hi Walter

I am currently working with 6gb DDR3 ram. Also, the machine is one of the new Nehalems, which as I understand have multithreading capabilities. I’d read about the concept of not using all the cores for a cluster, but that this did not apply to multithreading machines, which effectively create extra virtual cores. I’ll try and do a test using 3 cores later today

Cheers

Franco

London based Avid/ FCP offline editor.
FCP Online.

[Franco Bogino]

I tried with 3 instances in the cluster and there was no real difference, the render time was 3:55, so slightly slower. I also ran the render with activity monitor open, and for some reason it shows 8 cores running (I don’t know if this has anything to do with multithreading?), It showed all eight running at around 80% when using a 4 cluster, and at around 60% for a 3 cluster.
I have had one response on another forum stating that the stitch time for merging the multiple segments of the render does offset the faster render time. This means that there is a sweet point where the time gained is more than the stitching time. The poster suggested that with hour long sequences you really see the difference. I’m waiting to see if anyone else confirms this.

Thanks

Franco

London based Avid/ FCP offline editor.
FCP Online.

[Walter Soyka]

[Franco Bogino] “I have had one response on another forum stating that the stitch time for merging the multiple segments of the render does offset the faster render time. This means that there is a sweet point where the time gained is more than the stitching time. The poster suggested that with hour long sequences you really see the difference. I’m waiting to see if anyone else confirms this.”

It might not just be stitching; do you see a performance change if you set Compressor > Preferences… Cluster Options to “Never copy…”? (This would break over a networked cluster, but should work on a virtual quick cluster.)

Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog – What I’m thinking when my workstation’s thinking
Creative Cow Forum Host: Live & Stage Events

[Mark Spano]

Where is the source file and where is the destination? I have found only best results when performing the compression tasks on a file located on internal SATA drive, exporting to internal SATA drive. External drives, no matter how fast the access, seem to make the process slower. My setup: on an 8-core, I use 6 instances. Do not include unmanaged processes from other computers.

The other issue at hand is type of compression. QuickCluster works very well with H.264 and MPEG-2 – HDV may be another story. Try internal drives and if that doesn’t change the results, try different source and destination codecs.