Thanks for your comment that a small amount of aliasing is to be expected as it seems to confirm my findings, so I will stop searching for a miracle cure.
Now to the important point – the instability.
The workflow you outlined is identical to mine ……. Native HDV edited in the timeline, Export>QuickTime Movie using current settings and making a reference movie.
There are then at least 3 possible workflows from that point:-
1. If that reference movie is encoded and burned in either DVDSP or iDVD (using any combination of settings) the resulting DVD displays bad judder/jitter/shimmer wherever there is movement (either subject or camera).
2. If that QT Reference Movie is encoded in Compressor before being burned in DVDSP the results are OK.
3. Alternatively, the method I normally use is to omit the Export>QT Movie stage and instead Export>Using Compressor. This is a simpler workflow and gives the best results. I have a feeling that they may be a tad better than Workflow 2 but it is too close to be definitive.
So as long as a QT Movie (created from an HDV timeline) is encoded in Compressor, the results are OK, BUT try encoding that QT Movie in DVDSP or iDVD and bad instability is created.
Conversely, the OPPOSITE findings occur if I edit HDV that has been down-converted to DV in the camera during Capture.
In this scenario, when exported as a QT Movie and encoded and burned in either DVDSP or iDVD the results are almost perfect …… even the slight aliasing has gone.
However, encode it in Compressor and all the instability is back.
I cannot explain these apparently contradictory findings as I don’t understand in detail the inner workings of the various apps.
Last year I appeared to be the only one experiencing these things (always a bad sign!) but since then other people have reported similar findings.