Faster… Faster! (How can I make it faster?)

[Walter Biscardi]

[David McClave] “I need faster rendering in FCP, Compressor, and Motion”

Then you need more processor power. The more processor power, the faster the renders. Faster RAID will only help marginally. For Motion and Color, you need a beefy ATI graphics card. For FCP and anything else, you need faster processors. So that would be the fastest Mac Pro that’s out there right now will render the fastest.

Render speeds also depend on the format you’re working with natively and what you’re rendering to for finish.

Walter Biscardi, Jr.
Editor, Colorist, Director, Writer, Consultant, Author.
Credits include multiple Emmy, Telly, Aurora and Peabody Awards.
Biscardi Creative Media

Creative Cow Forum Host:
Apple Final Cut Pro, Apple Motion, Apple Color, AJA Kona, Business & Marketing, Maxx Digital.

Read my Blog!

[Alan Okey]

If you haven’t done so already, set up a cluster in Qmaster using all available computers, connected via gigabit Ethernet. Compressor takes advantage of distributed processing, so rather than having your spare machines sit idly by when you use Compressor, put the extra horsepower to work by utilizing Qmaster.

I’ve had good luck using managed clustering, setting up the host machine as the cluster controller and setting slave machines to “services only.” I recently set up a cluster for an editor friend who was going to buy a used dual G5 for $850.00 to increase her rendering speeds (she has a first-generation dual G5), until I explained to her that adding a new Mac Mini and setting up a cluster with Qmaster would not only save her money, but would run circles around simply upgrading to a slightly newer (and still outdated) G5. She’s quite happy with the results. The new Mac Mini absolutely trounces her G5 in rendering times – it’s easily twice as fast. While the Mac Mini is not a good choice for a dedicated editing machine, it’s hard to beat for cheap CPU horsepower in a cluster. The caveat is making sure you connect everything via gigabit Ethernet and have plenty of RAM.

Unfortunately, FCP doesn’t utilize distributed processing, so a cluster won’t improve render times within FCP. But for any Compressor jobs you might have, including compressing to web, DVD or doing format conversions, putting all of your available computing horsepower to work by using Qmaster will benefit your render times considerably.

[Alexander Kallas]

Very interesting post Alan,
Can you please post a clear description of the steps in setting this render farm up, I have seen many different versions, not sure if those were correct.

Cheers
Alexander

[Alan Okey]

Steps for setting up a Qmaster cluster:

1. Make sure all machines are connected to each other via gigabit Ethernet. GigE switches are cheap – pick one up if you don’t already have one.

2. Install Qmaster and Compressor on all machines. While it’s necessary to have a separate license of Final Cut Pro on each machine linked together on the same network, I don’t know if this is the case with Compressor. I don’t know if Compressor snoops the local network to check for other systems using the same serial number, or if it has an unlimited render node license. The systems I have set up have used unique serial numbers.

3. Open the Network control panel in System Preferences. Unless all of your cluster machines are connected to the same router with its own IP address, Choose DHCP for the network interface that you will be using to connect the machine to the cluster. Click Advanced, and check the box marked “make AppleTalk active.” This step may not be necessary, but without a router it may help the machines in the cluster to see each other more easily. If you do have a router with a dedicated IP address, you can set up your machines using static IP addresses and enter the router’s IP address in the router field.

4. Open the Sharing control panel in System Preferences. Make sure File Sharing is checked on. Click the plus button under Shared Folders and add the system drive of the machine. Click the plus sign under Users and add the username and password of the primary user account that you use on what will be the cluster controller, most likely your primary editing machine. Change user permissions for this account to Read & Write.

5. On the Setup tab of the Qmaster control panel in System Preferences, select the following for all machines except the primary workstation, from which you will be submitting batches – the primary machine will be the cluster controller.

A) Under “Share this computer as:” choose “services only.”
B) Under “Services,” check “Share” and “Managed” for the Compressor service.
C) Click “Options for selected service” and choose the maximum number of instances for Compressor.
D) Switch to the Advanced tab and under “Extras,” check “Show Qmaster service status in menu bar” and type in a unique name after “Identify this computer to Qadministrator as:”
E) Click “Start Sharing”

Perform the above steps for all machines except your primary workstation.

On the primary workstation, most likely your editing system, make sure to share your system drive in the System Preferences Sharing control panel and add the username and passwords of the primary user accounts of each of the machines that you set up in the previous steps. Grant these accounts Read & Write access.

Open the Qmaster control panel in System Preferences. This time, under “Share this computer as:” choose “Services and cluster controller.” If desired, perform the same procedure for service instances as before.*

Follow the same steps as the previous systems under the Advanced tab.

Once all systems have been configured and are sharing services, open Apple Qadministrator on the cluster controller system (located in the Applications folder). Click the plus button under “Clusters” to add a new cluster. Give the cluster a name. Next, select your machine name under the Controller button in the Services tab.

In the Qmaster services browser in the lower half of the Qadministrator window, you should see a list of machines that are available to add to the cluster. Clicking the disclosure triangle next to the machine name will display the number of available CPUs on that machine.

Drag the machines (or CPUs) that you wish to add to the cluster into the empty list in the Services tab. When finished, click “Apply” and quit the Qadministrator application.

When you are ready to submit a job or jobs to the cluster in Compressor, click on the Submit button and instead of choosing “local machine,” choose the cluster that you created in Qadministraor. Compressor will automatically distribute the processing of any jobs when possible. For Quicktime movies, video content will be divided into segments and processed separately, then rejoined seamlessly into a single file once all pieces are finished being processed. Some files, such as audio, cannot be segmented and will be processed by a single CPU/service instance.

This method has worked for me. I can’t answer questions about using other methods (quick cluster, for example), and I can’t really help with troubleshooting if these steps don’t work in your specific case. All I can suggest is to read the manuals and participate in user discussion forums on the Apple website.

I would also suggest that any clustering setup that enables Read & Write access between machines be set up behind a hardware firewall for security. Assigning each machine a static IP address in the private 192.168 range would also add a layer of protection. A true network/security specialist would most likely cringe at the thought of allowing read/write access to the system drive between systems on a network, but that’s an entirely different topic. For real network security advice, consult a professional.

**Note: It’s possible to exclude your primary workstation from the Qmaster cluster even if it is the cluster controller. You may wish to do this if you want to send jobs to Compressor to be completed in the background while you continue to edit, utilizing the full CPU power of your editing system. Be aware that your editing system’s drives will continue to be accessed by the cluster if any source media for your Compressor job batch is contained on it. It is possible to set up centralized cluster storage on a machine other than your cluster controller, but any source media used by Compressor will need to be copied there or located there from the start. Cluster storage is beyond the scope of this discussion. For further details, RTFM.

[Alexander Kallas]

Awesome!
Thanks, best description I have seen.

Cheers
Alexander

[David Bogie]

[Alexander Kallas] “Thanks, best description I have seen.
“

Agreed, thanks, Andy, very helpful and a great resource for future Fuster Cluckers.

I’ve got to say though, the whole process is ridiculously convoluted, very non-Macish. I’m dreaming, I now, but setting up and running Qmaster should be a one-button solution.

bogiesan

[Walter Biscardi]

[david bogie] ” I’m dreaming, I now, but setting up and running Qmaster should be a one-button solution. “

I gave up on QMaster because it’s just very flaky in our experience. When it works, it’s great, but more times than not, it just doesn’t work.

Walter Biscardi, Jr.
Editor, Colorist, Director, Writer, Consultant, Author.
Credits include multiple Emmy, Telly, Aurora and Peabody Awards.
Biscardi Creative Media

Creative Cow Forum Host:
Apple Final Cut Pro, Apple Motion, Apple Color, AJA Kona, Business & Marketing, Maxx Digital.

Read my Blog!

[Jerry Hofmann]

Except for the DV footage, get the rest of it to ProRes… HDV is really cumbersome to compress to anything… That alone will speed up your renders and your compressions. Qmaster works fine in the latest software here… and it’s consistently OK. But I NEVER edit in any GOP structured media because it’s so darn slow rendering and compressing. ProRes just plain rules here.

What you give up doing this is file size (Pro Res creates larger files than HDV does for sure)

Also exporting self contained QT movies to Compressor instead of sending directly to compressor from FCP is faster too… But there may be a slight quality loss. When you send unrendered sequences to compressor directly from FCP, the quality is best, but it may not be better enough for you to spend the time doing it.

Jerry

Apple Certified Trainer

Author: “Jerry Hofmann on Final Cut Pro 4” Click here

8-Core 3.0 Intel Mac Pro, Dual 2 gig G5, AJA Kona SD, AJA Kona 2, Huge Systems Array UL3D, AJA Io HD, 17″ MBP, Matrox MXO, CD’s

[Scott Norton]

I’m hoping you get emails when your older posts responded to. Your tutorial is welcome here. I am thinking about giving this a shot on my modest setup.

Quick question. If I only have 2 computers, can I connect Ethernet directly between my mac pro and MBP? Skip the switcher?

Thanks so much for all your help already.

[Alan Okey]

You should be able to connect the two computers without a switch.