“Quick Time Movies” You never said what codec you were trying to use. You may need to use another codec brand if quicktime is unfixable which means you can’t use quicktimeplayer’s export, only AE’s.
adding effects may be pushing AE over the edge from your high mem settings. try the defaults so that there is room for your graphics buffer to complete the effects.
max mem usage 120%
max ram cache size 60%