Unused plugins should not affect the performances… but if they occupy a lot of space in the HD they could slow down the entire system. I never fill the system HD more than 60%.
RAM/CPU/GPU are important but disk read/write speed is important too and affects the rendered/unrendered/effected/uneffectced clip playback.
In your case the bottleneck could be the HD performance. If you can, try a faster one.
Or maybe you are simply using too much or too heavy effects for your machine.