-
Compressor 4 “Render Farm” :: What is Required?
One of the reported advantages of Compressor 4 is the effective integration of Qmaster and the ability to more easily distribute renders to other Macs with Compressor 4 installed. I am trying to get this working and am having some trouble doing so.
I’ve tried setting up QuickClusters & Managed clusters, but the jobs are not distributing across the machines as I thought they would. I’ve looked at the C4 tutorials on Lynda.com, searched the forums here and scrubbed the Interwebs for example setups. I’ve read the docs as well and I am clearly not groking them or missing something crucial in my read.
The only thing I think could be a problem is I am not setting all of the nodes up with a central data store on the network. Each is working off of their local hard drive.
So, before I invest in a SAN, a couple of questions:
1) Have I misunderstood C4’s ability to create a video render farm?
2) If not, does distributed render in C4 require a central data store (CDS)? If not, what else might be the issue?
3) If the absence of a CDS is the issue, does it require a SAN? Or, can I used a mapped drive to a NAS? If a SAN is required, does it have to be Apple XSAN? Or, could I use something like Bob Zelin’s DIY SAN (link below).
https://magazine.creativecow.net/article/build-your-own-affordable-san-that-iworksi
A little about me, I am pretty technical. I build and modify machines all the time. I also have a pretty firm understanding of networking principles, but would not be considered a network engineer. I have experience with Episode Engine on both OSX & PC, so I know this can be done. I am just looking to do it using C5, or if needed C3.5.
In any case, feel free to be pretty technical in your responses. I should be able to follow you and ask good questions when I don’t.
Thanks,
Doug
PS: If there is another forum which is more appropriate for this conversation, please point me there. I am new to the COW, and just learning my way around the community.
-
41 Replies
-
My experience has been that you need a central data store. It can be an afp drive.
-
Thanks Ian. Will give it a go with that configuration.
-
hi Doug –
I cannot answer your question. I am responding here to your email to me on 5/27/12
A SAN is not a render farm. I do not know how to setup a C4D render farm. But I can tell you one thing. In the article that I wrote that you referenced, if you create a SAN like I described and turn QMaster on, and try to do this type of C4D render, it will lock up the entire system, and no one will be able to work. I had this exact example happen at a client with 5 workstattions (FCP), and one C4D workstation. They turned on QMaster on all workstations, and started a large render and within 30 minutes, everyone locked up.So it doesn’t work.
Bob Zelin
-
Bob,
RE: SAN is not a render farm. Understood. I was thinking of the SAN as attached storage for the render farm, as it seems that Compressor 4 requires a central data store to allow distributed rendering.
I plan to use Ethernet, probably link aggregated Gigabit as I am working with ProRes and not Uncompressed. So, if my research is accurate, it should give me the through out I need.
Having said that, I now suspect a SAN is overkill for my needs. Rather, I think I could do a DIY NAS an mount it to all the render nodes using AFP. In this case, I still plan to explore using NICs and a managed switch with link aggregation to get the maximum throughput across the “farm”. Any gotchas with that idea?
Thanks,
Doug
PS: Did I send you an email? I don’t think I have you email address. 🙂 If I did, I don’t seem to have it. In any case, thanks for taking the time to answer. We are exploring adding C4D to the mix, so your use case is very timely and valuable.
-
Wow. Just reread that. Lots of typos I missed. Sorry for my fat fingers.
In any case, the only one that may require clarification is “through out”. I meant “throughput”. I think auto-correct got me. 😀
-
For simple rendering and not necessarily streaming video, a managed switch is not necessary. While the cost difference may not be that high I’m just trying to save you a buck. You can just throw a NAS on your current network or use one of your systems to serve out a file share.
-
Have tried the AFP mount method and am having trouble. Here are the methods I have tried. In each case, I have all machines mapped to the same folder mounted locally over AFP.
1) Primary machine as “as Services and cluster controller” with other machines “as Services only”.
2) Primary machine as “QuickCluster with services” with other machines “as Services only”.
3) All machines as “QuickCluster with services”.
In each case, I submit a job using “This Computer Plus” and the job goes into the queue and just hangs. There is no movement or status updating. I even let it run overnight thinking it might need to copy some files. In the AM, it was still hung.
Here is a video walking through the setup and render.
https://screencast.com/t/Nv5kLnP0GTPx
What am I missing?
Thanks,
Doug
-
By the way, I have read am continuing to try to peruse the Compressor 4 online docs to find my misstep:
-
I would stick with actually creating clusters – I have never been a fan of that quick thing. YOu did not show in your video where the destination is – is that destination also on the AFP share?
-
Ian,
I figured going with a cluster controller would be the best option as QuickCluster is intended for making a cluster out of a multi-core machine. Here is a more detailed video, this time with audio, of the setup and the issues.
https://screencast.com/t/7DPoBFBm8x5
Thanks,
Doug
Reply to this Discussion! Login or Sign Up