If you haven’t messed up with the file frame rates then your problem is a bad dolly/slider/not so smooth operator. You might want to try the Warp Stabilizer plugin to smooth it out.
If you want better performance you won’t get it by dumping your machine full of RAM. You need CPU and possibly GPU performance. I’m not sure what GPUs the Red Giant plugins will use but I’d get one that does. Also not all effects will run realtime no matter how good your machine is.
Encore will transcode your files into standard definition if you try to make a DVD. If you wan’t an HD disc you need to get a blu-ray burner and make a blu-ray.