first off, you should never use ae to render to h.264 or mpeg-2 based codecs. render from ae at the highest possible quality codec (uncompressed, lossless animation, photo-jpeg, prores, etc) and use a compression utility to compress to those types of files.
but you mentioned that it crashed when rendering to ‘none’, so i don’t think the crash is related to the compression.
by your machine description i think it is a 2008 macpro. if i’m thinking of the right machine, the ram configuration of those macs is very specific, and you can not only get poor performance if it is installed incorrectly, but you can also get stability issues.
see this page for some info on configuration:
https://www.barefeats.com/harper3.html
if you have the manual, that will help too.
you can find the current configuration for your ram by choosing ‘about this mac’ from the apple menu and clicking the more info button and selecting ‘memory’ from the hardware properties. then compare what you see there with what you find in the manual or that webpage…
Kevin Camp
Senior Designer
KCPQ, KMYQ & KRCW