Have you tried purging every x number of frames in the secret prefs? If indeed this is a memory issue it might help.
To access the secret prefs press the Shift key as you select Preferences > General from the menu.
I see an earlier post from you relating to Motion Blur with the same project, depending on your settings this could demand large memory resources to render, particularly since the nested Comp is 6000 X 6000 pixels.
