I had the same problem recently and the reason it happened for me was because when i applied the Roto Brush my preview wasn’t in full quality it was in half. The only solution i found to fix this was rendering the final video in half which i didn’t want todo so ended up redoing it