What plug-in are you using? Also – has your image shifted at all in either x or y? If you import your footage and apply a different plug-in do you experience the same problem??
I notice that certain plug-ins when applied definitely have an effect on the image regardless of the frame-rate. I have this problem all the time with trapcode shine – even when ray length and boost light and all other attributes are completely off the image becomes noticeably blurred. Other Boris Plug-ins and Eye Candy plug-ins have given me the same problem. My workaround in the case of shine has been to apply it to an adjustment layer and then trim the adjustment layer to only the time when the rays are ramping up or down only.