Here is how to get rid of this damn problem that makes everyone crazy. A little late, but it might help people anyway.
1. Hit and hold SHIFT in AE
2. Click on Edit > Preferences > general, (then only release shift) and you should see “SECRET” at the bottom of the preferences list. Check “Do not cache layers” and, above all : clear image cache when rendering, every 50 frames (more or less).
This will regularly purge the video memory while rendering, and you will notice your RAM usage making sawtooth instead of insanely increasing.
Spread the word. It is absolutely ridiculous for such a thing not to be enabled by default, and worse, that it is hidden in a super-mario secret menu. There are some dumb guys in Adobe’s team.
P.S. After Effects is BY FAR the most unstable software I know, it keeps crashing and crashing all the time for any reason, and it did so with ALL versions I tried (from 5.0 to CS5) . A CTRL+S every 3 seconds is absolutely essential when working with this crap. This is even more surprising when I see how other Adobe’s products (especially Photoshop) are rock-stable.