jagged edges on clips once you move them in 3D is a fact of life with FCP 4.5 and below…this has been drastically improved in fcp 5 from what i have seen.
the only way to get around the jagged edges is to use a 3rd party app like after effects or boris red.
if the video you are moving with the basic 3d filter is against black or another solid colour you could try masking your video, making sure the mask is the same colour as your background to hide the mask. could be a way to hide the jagged edges.
the artifacts you are seeing will look worse in your canvas than if you previewed to a broadcast monitor – as noted above by gunner.
Not that i work for apple or anything but the general improvements in version 5 are well worth the upgrade IMHO.
system 1 G5DP 2.0 / 2.0G RAM / OS 10.3.8 / QT 6.5.2 / FCP HD 4.5 / CineWave 4.7 / Cinewave RT Pro/ Pro Digital Plus & Pro Analogue BOB