I had a similar problem with Keylight…although my problem was that it literally cut the layer in half for some frames. It would only happen intermittently and when it did, it was devistating. I even had a drop shadow on the layer, and the drop shadow shadowed the part that was cut off.
My video was exported from Avid Xpress Pro – both as reference files and as regular exported quicktimes.
-Dan