I’ve had a similar issue, where file formats that require quicktime to process (such as .mov) will fail with all kinds of weird issues, when I’m running arender from my custom software service, while everything works fine when running the AE gui or the aerender command from the administrative console.
The problem was eventually identified as missing system environment variables. You must have the following environment variables defined in order for After Effects to successfully call quicktime:
PUBLIC
WINDIR
PATH
Apparently if only some of them defined, quicktime might still work, but I recommend setting them all up. You can use the values as can be seen by running ‘dir env:\’ from your powershell prompt, though for PATH I recommend using a minimal value including only the Windows system folders and the quicktime folder.