I experience exactly the same thing.
MacPro 1,1 (yes, it’s time for an upgrade), 12 Gb RAM, 4 left for other apps (even though AE is the only one running at the moment), multiprocessing turned off.
Every mouse click means waiting half a second or I’ll move whatever I just clicked.
But… This only happens when using the mouse. With my Wacom a click is simply a click as it’s supposed to be. (problem solved? No. My Wacom is set to work only on the main screen while my timeline sits on the left screen. Spreading the Wacom to both screens will make the Wacom useless for precision paint jobs).
First I thought is was my slow machine, but since the Wacom works OK there must be something else…