-
AE network rendering
Hi – I hope someone can help.
I’m still running MacOSX 10.3.9 and AE 6.5 because I just can’t afford the accumulated upgrade costs of 11 Macs and all associated software.
I have a render farm used for ElectricImage Renderama and I also use this for AE Watch Folder network rendering. This works fine EXCEPT …
… 5 slave machines handle the watching process with no problem and pick up immediately on any job being dumped into the watch folder – but 4 other slaves ignore it. I have to restart each of them, mount the shared disk drive, launch AE RE and select the watch folder again before these 4 will recognise a waiting job and join in.
And this needs to be done while a job is actually in progress. If a second job is waiting they will roll over onto that as well, but the moment all jobs are complete they are effectively useless for any future jobs appearing in the watch folder – until I restart them all.
They are all set up identically and I just can’t figure out what is different about these 4 except that they are all on a single monitor switch and the others have their own monitors. But I can’t believe that could have any effect.
I’ve run out of things to try – has anyone experienced anything similar or got any ideas I can try please ?
Many thanks
Rod
-
6 Replies
-
Would any of these machines start the render at all if the working 5 machines are off?
Do any of them create any meaningful log entries?
Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog – What I’m thinking when my workstation’s thinking
Creative Cow Forum Host: Live & Stage Events -
Thanks for coming back so quickly on this Walter.
Spurred by your question I have just carried out some more tests.
1 – I launched just one of the 4 in question on its own and set it to watch the folder. When the job arrived in the watch folder it picked it up and rendered it as I would expect. I left it watching the folder for the next job.
2 – I then launched the remaining 3 of the 4 in question and set them all to watch. When a job arrived in the watch folder the latest 3 picked the job up but the first that was still watching did nothing but apparently keep on watching.
3 – When a third job was sent to the watch folder none of the 4 recognised that a job was waiting and all simply continued their count-downs.
4 – I restarted all 4 machines and re-launched RE and set the watch folder. I then added 4 of the other slave machines and set the watch folder on them. When a job arrived in the watch folder the 4 in question didn’t pick it up but I noticed in the search cycle that a series of messages were appearing at the end of each count cycle – .DS_store – Test Folder – test_00.htm – No Renderables found. After a couple of cycles this stopped and the search count down simply rolled over. The other machines were at this stage only half way through the job.
5 – As the test composition was only 100 frames and going through extremely quickly I changed it to 300 frames and sent that to the watch folder. For some reason now all machines picked it up.
After a pause between each with all machines just watching I have sent a further 2 jobs and all machines picked them up both times.
So for some completely inexplicable reason the system seems now to be working just as I would expect it to !
But I hate not understanding WHY. It’s a bit like the sort of Voodoo of Scsi in times past.
Anyway – thanks for whatever magic you just performed !
cheers
Rod
Rod
-
[Rod Lord] “Anyway – thanks for whatever magic you just performed !”
Ha — happy to help!
[Rod Lord] “I noticed in the search cycle that a series of messages were appearing at the end of each count cycle – .DS_store – Test Folder – test_00.htm – No Renderables found.”
This strikes me as the biggest clue to what’s really going on. What machine/OS/file system is serving the files? Do you have a cross-platform workflow? What are the paths and filenames like? How long does it take Finder on a render node to “see” that the shared storage has been updated with new files?
Also, I noted you’re running AE 6.5 on 10.3.9 — which means you’re still on PowerPC hardware, right? I just dusted off a couple old benchmark projects and fired up a G5, and you might find the results interesting.
Dual-core 2.3 GHz G5 (2005) with 12 GB of RAM running AE CS3 crunched:
- Night Flight — 13:19
- Total Benchmark — 15:47
8-core 2.8GHz Mac Pro (2008) with 32 GB of RAM running AE CS5 crunched:
- Night Flight — 1:40 (7.99x faster than the G5)
- Total Benchmark — 1:52 (8.45x faster)
My Mac Pro is still a powerful workstation, but it’s no spring chicken. I’d expect still better numbers from the 2009 and 2010 models. Taking that into account, as well as all the network overhead that Adobe’s naive network watch folder system requires, and I think that a well-equipped, modern Mac Pro running CS5 will give your 11-node PPC render farm a run for its money.
Also, AE CS5 is the first version of After Effects I know of which has restricted the legacy projects it can open. CS5 will only open projects saved in AE v6.0 or later. This may be a consideration for your studio going forward as you plan your upgrades.
Walter Soyka
Principal & Designer at Keen Live
Motion Graphics, Widescreen Events, Presentation Design, and Consulting
RenderBreak Blog – What I’m thinking when my workstation’s thinking
Creative Cow Forum Host: Live & Stage Events -
I’m sure you’d checked this (probably repeatedly) but when you do a ‘collect files’ before a batch render, AE by default sets “Maximum number of machines” to 5, coincidentally the number of your machines that DID pick up the jobs and render them.
-
[Walter Soyka] “Do you have a cross-platform workflow?”
Absolutely not ! Ever since I retired an old IBM as a terminal on a Prime mini back in 1992 and replaced it with a Mac IIfx I’ve successfully avoided having anything Microsoft within corrupting distance. I’m not sure that’s a valid viewpoint these days but I’m too old to change now.
[Walter Soyka] “How long does it take Finder on a render node to “see” that the shared storage has been updated with new files?”
Well – it seems pretty well instant. Those nodes that pick up seem to recognise that something’s changed within milliseconds. But they do struggle with each other to load the project at the same time.
[Walter Soyka] “a well-equipped, modern Mac Pro running CS5 will give your 11-node PPC render farm a run for its money”
I’m absolutely certain you’re right but just at the point when I was looking at upgrading my major clients started to retire … or die !
So it’s become an increasing struggle to justify the necessary investment out of the diminishing budgets. Shrinking client base on top of a rather long recession. As long as all this stuff keeps stumbling on and delivering I’ll stick with it. The crunch of course comes on the day the old dual G4 workstation collapses – when I will have no choice.
Thank you so much for all the trouble you’ve taken on this. I just wish I now understood what was going wrong before. But as long as it keeps working now I’ll just accept that it’s voodoo.
Rod
-
[Paul Roper] “I’m sure you’d checked this (probably repeatedly) but when you do a ‘collect files’ before a batch render, AE by default sets “Maximum number of machines” to 5, coincidentally the number of your machines that DID pick up the jobs and render them.”
Thanks Paul
Yes – it was the first thing I double checked as I immediately assumed this was the problem exactly because the numbers were co-incident.
But on the very first job I sent I had changed that number to 10 and it’s remained the default ever since.
There’s some other sort of voodoo been going on which I don’t understand but it seems to be working OK now so I will just keep my fingers crossed, knock on wood, etc, etc, – and keep ploughing on.
Thanks for the thought.
Rod
Reply to this Discussion! Login or Sign Up