I run into this all the time in CS3. I believe there is a serious memory leak in CS3. I notice after working for various amounts of time I will get this error. I, too, have tried every combination of RAM cache setting I can think of & purging the cache & emptying the cache, ect. Over time it simply fills up. Eventually even clearing the cache will not help & the program MUST be restarted.
To further complicate the matter & alienate me from CS3, is the fact that the problem extends to renders. God help you if you work in HD 1920x1080i. Turn on 3D layers & a single light & roll the dice if it will render a single frame. I have 4GB of memory but that is useless since AE only can use 2GB no matter how many gigs you have.
I have an HD comp with 7 3D layers, 1 camera, 1 light. This is a precomp I’d like to use. It’s not complicated at all. Basically it is 2 1920×100 psd files, 1 720×480 psd, 4 solids, the camera & the light. With shadow cast enabled it cannot render a single frame. Turn off shadows & I can get through a second or so before failure. Restart another few frames & so it goes. Turns out I don’t like it without the shadows & now must keyframe back in the shadows. This turned an afternoon into 2 18 hour nightmares. I’m questioning AE as a HD tool. I don’t have the fastest machine but I think I have a decent machine. A Dell 4600 with 4 GB RAM.
If anyone can think of something I’m missing I’ll try anything. Right now I am looking at redesigning entire projects as I move forward to avoid lights & cameras. Adds a hell of a lot of work faking those things & they don’t look as good.