Okay, so it was the Motion Filtering Quality…here’s what Dave Simon had to say…
The issue of the jagged edges has been RESOLVED! Whew. It is that damn Render Filter setting in the Video tab of the sequence settings. The funny (read: not funny) thing about that setting is that when you open a project that was done in 4.5 with FCP 5, FCP goes through a conversion process. In that process, it asks you want kind of filter you want to use (Best, Normal or Fastest). Obviously, we want Fastest for two reasons. One: We’ll eliminate problems similar to what I just went through with jagged edges. Two: The conversion dialog box that you are presented with when opening an old seq in 5 actually tells you that the old sequence was done with the Fastest filter! “Yeah, we know you did it this way before, but what about trying something new?” WHAT?