I had the same problem. Everything else than qt worked fine, with qt you couldn’t even change the settings (small windows poped up and disappeared in a fraction of a second).
This was a new setup, fresh OS (snow leopard, QT7 installed as well), fresh CS5 install, 8-core….
Even new scene with standard setting couldn’t render out in Quicktime.
Did some testing today and found the solution. The problem on my system is Optical Flare by videocopilot. If you initialize it once, afx won’t be able to render out to quicktime. At least this is the case with CS5 on Snow leopard.
What to do….:
Remove all the optical flare plug-ins from your scenery and save.
Go to User/Library/preferences/adobe/After Effects and delete all prefs.
Start afx and it will be able to render out qt as expected.
BUT, if you open/active optical flare ONCE, you will have the same problem again. If you need to use it, render out in e.g. Tiffs. After that, quit afx, delete prefs….and you will be able to render qt again (but not if you open an optical flares accessing scene.
hope that helps.