This has been an issue with many CS5 installs on the Mac.
Since CS5 is 64 bit, but uses a 32 bit version of QuickTime, Adobe initiates a process called QT32 Server.
If this file (QT32 Server) isn’t in the correct location, or your Firewall settings don’t allow it to communicate with After Effects, it can result in the error you’re experiencing.
Sometimes repairing permissions fixes it.
There are several solutions depending on the precise nature of the problem. This has been covered (and continues to be covered) at the Adobe After Effects forum. Many After Effects users have had this same issue.
Check this out: https://forums.adobe.com/message/3309646#3309646
Look to the forums there for other solutions, there will be one that works for you.