This might be a little late, but possibly useful for future reference; I’ve experienced the same problems with crashes during renders as well as the “out of memory” error (despite having 8 GB of ram on my system). The problem seems to be related to how fcp treats cached video during renders – where e.g. certain fx-script effects or plugins (or combinations thereof) inhibits fcp/osx in flushing/releasing inactive ram, which causes fcp to “run out of” free ram where it can temporarily store the video render data (and then either crash or display the “out of memory” msg).
A workaround for this is to run the (system) activity monitor (under applications>utilities) while rendering in fcp, and closely watching the inactive memory usage under the system memory tab, and abort the render when the inactive memory (the blue slice of the cake diagram) closes in on 90-95%, then save the project and start the render over again. Keep on doing this until the render is done. It’s a bit more work, but a lot better then leaving an overnight render just to discover the next morning that absolutely nothing has been rendered..
I’ve run into this problem more often with higher resolution footage eg 1080, and even worse with large jpegs that has to be scaled for ken burns effects or similar. Also the the (boris)”title 3d” generator is notorious in producing these kinds of errors..
Hope this will be a thing of the past though, when fx-plug architecture slowly replaces the fx-scripts..