workflow questions regarding dv/uncompressed

[Mark Raudonis]

BabaG,

Others can correct me if I’m wrong, but I think you’re misunderstanding the “when and how” of working in an uncompressed format.

You shot on DV. It will NEVER get any better than that. The point of working in an “uncompressed” format for finishing is to MAINTAIN quality to the end. You don’t have to “import” or media manage anything.
This would be my workflow. Cut in a DV timeline. Lock picture.

Create a new “uncompressed 8 bit” sequence. Drop your “locked” DV sequence onto this 8 bit sequence. Poof! You’re now working in 8 bit. No need to export, reconnect or anything. You may have to render to play, depending on your processor speed. Do your color correct, GRFX etc. in this uncompressed timeline. Create your “uncompressed” QT or go out to digibeta tape to maintain your quality. If you’re starting in DV, using a 10 bit timeline is just a waste of time. I’d suggest 8 bit. You’ll never see the difference. (Unless you’re adding alot of GRFX)

It’s that easy. Good luck.

Mark

[Walter Biscardi]

If you take your DV footage into an Uncompressed timeline, then record it back to DV, you’re actually going to end up with a worse image than if you had just left it in the DV workspace all along. Everything will be recompressed 5:1 on top of the original DV 5:1 compression.

Best way to get DV to Uncompressed is to get a capture card like the AJA Kona boards or the AJA Io external boxes and simply capture your DV as uncompressed on the way in. Then master out to DigiBeta or something like that to maintain your uncompressed quality.

Walter Biscardi, Jr.
https://www.biscardicreative.com
HD Editorial & Animation for Food Network’s “Good Eats”
HD Editorial for “Assignment Earth”

“I reject your reality and substitute my own!” – Adam Savage, Mythbusters

[Baba Goof]

the idea here was not to make anything better but to
keep it from getting worse. the piece in question has
a lot of subtle layerings and superimpositions lasting
most of the duration of the piece. it’s an art pice.
the idea was to maintain image quality through the
process of rendering and processing these supers into
the single composited image.

the idea of media management was more to do with disk
management. the filmmaker does a lot of trial and error
and his timelines are littered with lots of media that
never actually gets used. he’s also not particularly
savvy about management and workflow so things can get
messy. in that context i wanted to use the media manager
to consolidate a simpler project containing only the
final material, minus all the experimentation, from
which we could derive our masters, maintaining as much
quality as possible.

good to hear that 8bit will suffice. in context of the
layering i’m describing, does the comment about dv—>
uncompressed—>dv being worse because of recompression
still hold? i’m still not clear on this. makes sense in
context of an unaltered image, but not clear on whether
this still applies when a new image, one comprising
three or four full screen supers, is being created. iow,
does it still hold when the image being rendered is
significantly different than any one of the originals?

thanks,
BabaG

[Walter Biscardi]

[BabaG] “good to hear that 8bit will suffice. in context of the
layering i’m describing, does the comment about dv—>
uncompressed—>dv being worse because of recompression
still hold? i’m still not clear on this.”

Yep. What you’re going to do is create an 8bit uncompressed timeline. You will then need to re-compress it to DV to output to the DV format. Either through FCP using a DV timeline or playing directly off your uncompressed timeline to a DV deck. All of your video will be compressed a second time into the DV codec and all of your uncompressed graphics will be compressed into the DV codec for the first time.

If your final output is DV, stay in the DV codec. This way you can deal with artifacts and any other graphics issues while you’re working. If you work in uncompressed then lay off to DV tape, you may see a lot of unwanted artifacts in the graphics that you would never see in an uncompressed timeline.

Walter Biscardi, Jr.
https://www.biscardicreative.com
HD Editorial & Animation for Food Network’s “Good Eats”
HD Editorial for “Assignment Earth”

“I reject your reality and substitute my own!” – Adam Savage, Mythbusters

[Bill Lee]

I’ll agree with Walter on the recommendation to stay in DV, but I’ll have to disagree with Walter on the main reason why (this is possibly a very bad move and may show the world the extent of my ignorance, but I’ll just grit my teeth and continue). I still could be wrong and Walter’s explanation might be the dominant effect on the quality of the final video.

I believe that the two biggest reasons not to change the video from DV to Uncompressed are: a) the difference in field dominance between the two DV and Uncompressed Presets for PAL and b) the different frame sizes for the NTSC DV and Uncompressed Presets.

Video compressed with an uncompressed codec is… well, uncompressed. It has no inherent problems in codec compression artifacts since it doesn’t compress the video. It will however retain the artifacts of its source material or applied filters and processing. I’ll be analyzing the four specific types of uncompressed Preset supplied by Apple: 8-bit Uncompressed NTSC 48kHz, 8-bit Uncompressed PAL 48kHz, 10-bit Uncompressed NTSC 48kHz, and the 10-bit Uncompressed PAL 48kHz. There are other types of uncompressed video possible, but the supplied Presets have particular characteristics that need to be examined.

DV on the other hand has a 5:1 compression of the video – every time you have to recompress it, it will add additional artifacts into the image. The DV decompression is another possible source of artifacts, but you can’t escape doing this at least once (since you have to extract the data), and after that you can lump the compression artifacts and the decompression artifacts into one pile if you are going to do a couple of rounds of compression/decompression cycles. On the other hand, multiple DV compression is not that bad, and certainly not as bad as n-generation analogue tape copies. If you have DV already, you have already taken the major hit on quality – it doesn’t get too much worse than that for one or two re-encodes. I’m assuming that you’re using a reasonably good DV decoder/endcoder here.

So, to get the best possible image, you start off with DV, you have it decompressed, you do things with this video and avoiding compressing it back into DV right up until you have to put it back out to DV tape. It would seem that on the face of it, an 10-bit Uncompressed video format would make a marvellous intermediate format to maintain the quality while you put the video through various processes. Unfortunately it is not, for reasons of Field Dominance in PAL, and Frame sizes in NTSC due to the particular natures of the Uncompressed Presets supplied with FCP.

Field Dominance
————–
Your DV video is 59.97 (50 in PAL) fields per second, with half of the image captured and shown in every field. Both DV PAL and DV NTSC have lower field dominance, which means that the 1st field is shown first and the 2nd field is then shown (the 1st and 2nd are not refering to when they were captured, but how they are written to the media). If they are shown in the wrong order your picture will look poorest in the areas of motion inside the frame. Apple has an article on this at (docs.info.apple.com/article.html?artnum=58634). The supplied Uncompressed PAL Presets are upper field dominant, so if you bring your DV PAL video into an Preset uncompressed PAL sequence, the size of the frames are the same – 720 x 576, but the fields will be played back in the wrong order and so it will appear to have gross artifacts. Filters that use the temporal difference of 1/50 second between alternate scan lines to generate their results will be incorrect. You could shift the frame down one pixel to fix the field dominance, but then you have just destroyed a scan line at the bottom, and now you have an extra one at the top that needs to be cropped or synthesised, or plain ignored.

NTSC Frame sizes
—————
Since both Uncompressed NTSC Presets have a Lower Field dominance, and DV NTSC also has a Lower Field dominance, then we would expect that we wouldn’t have the same field dominance problems as you would with PAL. There is a problem with the frame sizes though, and the field-based nature of video does cause problems in more subtle ways. DV NTSC has a frame size of 720 x 480. Uncompressed NTSC has a frame size of 720 x 486. When you place the DV NTSC clip into an Uncompressed NTSC timeline, you don’t get it scaled, since the inserted clip is smaller than the timeline frame size, but it is inset into the frame at its natural size and shifted down three pixels (scanlines) to center it in the frame. This has the effect of changing the field dominance, and so the NTSC video acquires the same field order problems as PAL has above. In addition, when the uncompressed sequence is converted back to NTSC DV, the 486 scan lines are compressed back to 480 scan lines, leading to a mixing of the scanlines during the interpolation and thus the destruction of the separate temporal nature of the fields in the frame, further damaging the quality of the video.

Testing: I created an DV NTSC video clip with frames consisting of two fields one of black scan lines and the other of white. This is the worst case scenario, and thus should demonstrate it quite nicely. I imported this into an 8-bit Uncompressed D1 sequence, and it centered the video, down three scan lines from the top. I then exported this video back out as a DV NTSC video clip, and viewed it in the QuickTime Player, ensuring that the High Quality option for video was turned on. There you can see of the deleterious effects of the scaling, especially when compared to the original DV. Secondly I created an animation in Motion, consisting of a white ball about 10% of the height of the frame on an NTSC DV timeline with a black background. I animated this diagonally left to right, and exported this out with field rendering turned on, but motion blur turned off. This rendered clip was imported into the same Uncompressed DV used above and played back at 100% in the canvas. Each field was clear and sharp, demonstrating the comb effect of interlaced video on the left and right edges of the white circle. I then exported this Uncompressed NTSC sequence as a DV NTSC movie, opened it in QuickTime Player, turned on High Quality and looked at the video. There were obvious scaling issues.

I also tested a similar animation in PAL DV inside a PAL Uncompressed sequence, and it took a while to turn off all the settings designed to save our eyes from flicker on my external Digital Desktop preview. I tried to see if my PAL monitor would show the difference between imported PAL DV and imported PAL DV shifted down one pixel, but couldn’t see a difference in the two clips. This test therefore remains inconclusive, and if somone with a better setup and broadcast monitor could make this test and report back as to whether there is an actual difference in motion artifacts due to field order display.

Notes about the uncompressed and the DV codecs:
——————————————-
The default Uncompressed PAL and Uncompressed NTSC formats offered by FCP have certain characteristics that make them unsuitable for direct use with DV clips. You can make your own Uncompressed format that would be kinder to your DV, matching the frame size as well as the field dominance to avoid the problems I’ve discussed above. The question is: Is it worth it? Let’s first have a look at the issue of loss of quality from encoding/decoding losses.

When you place a DV-codec clip in an 8-bit Uncompressed sequence, the video will require decompression before being played back and it may require rendering to play it back in real time. The render file will be uncompressed video, and should only suffer from round off error in being decompressed. If your uncompressed sequence is 10 bit, then your round-off error is a 1/4 that of the 8-bit error, bearing in mind that your source DV video will only give you 8-bits of information and you can’t retrieve quality already lost: you can only try to maintain what you have left. If you apply a transition, filter, or colour correction to that clip, the source DV is decoded again, the calculations made, and the render file generated anew. If you have nested sequences inside that sequence, any render files from the nested sequence timeline will be used instead of the original files. Too deeply nested and rendered sequences can be a source of accumulated round-off error but won’t suffer from lossy compression accumulation. You can force the original source files to be used instead of the render files by using the Render Manager to delete the intermediate render files of the nested sequences, and then further decrease the error in calculations by turning on the High Precision YUV in the Video Processing Sequence Settings. The number of DV re-encodes can still be just the same as in a completely DV-based sequences, and the intermediate uncompressed codec is not nearly as subject to quality drift if you are using intermediate render files from nested sequences. As Walter pointed out, viewing your uncompressed video on an external monitor is not going to allow you to make accurate decision on how the final product will look, as it will need to go through another DV encoding before it is ready.

When you place a DV-codec clip into a DV sequence you should want to end up with a DV-codec encoded output, otherwise you may be doing an unnecessary DV encoding. The DV still has to be decoded to an uncompressed format, it just happens ‘underneath the hood’ where you can’t see it. If you have turned on High Precision YUV in the Video Processing Sequence Settings, then all the filters, transitions and adjustments are done in high precision calculations. The use of nested render files will cause generational quality loss, so you might want to remove these intermediate render files before a final render, so that the DV-encoded video is decoded once only from the original DV encoded media. Then the video is encoded back as DV, maybe as the final version to go back to tape or may be re-encoded in another format. If you have to work on the video in other applications, then the need to export as DV and later reimport as DV might be a reason to use an intermediate lossless compression codec such as PNG: see(lists.apple.com/archives/quicktime-users/2005/Jul/msg00200.html) for its advantages over the Animation codec. You gain the biggest time and quality advantage if your video requires no processing at all, in which case the output video data is exactly the same as the input video data. If you have nested DV sequences, just before you are finished with the project, delete the intermediate render files and just generating render files for the outermost sequence will maintain the highest quality video.

The other question is: Can my playback system handle my custom DV-compatible uncompressed video, because if it can’t then you’ve created a workflow that is far less efficient than a DV-only workflow. Even then, the non-standard nature and tweaking required to make the DV-uncompressed workflow successful make it less attractive.

With care, the DV-only workflow can be just as accurate as an DV-Uncompressed workflow, if you are not round-tripping the video with another application, and remains a simple and well tried workflow with predictable results.

Bill Lee
“Darn, that took a long time to test and write up”

[Alexander Kallas]

[billlee] “NTSC Frame sizes
“Testing: I created an DV NTSC video clip with frames consisting of two fields one of black scan lines and the other of white. This is the worst case scenario, and thus should demonstrate it quite nicely. I imported this into an 8-bit Uncompressed D1 sequence, and it centered the video, down three scan lines from the top. I then exported this video back out as a DV NTSC video clip, and viewed it in the QuickTime Player, ensuring that the High Quality option for video was turned on. There you can see of the deleterious effects of the scaling, especially when compared to the original DV”

Hi Bill,
This test demonstrates exactly what we should NOT do, ie re-encoding to the same codec. Why would we ever go back to DV?
In your post you mention that DV artifacts will not show when > un-compressed (they are hidden). I assume we go un-compressed only when COMPLETELY finished (text aside), and then to delivery codec. DV is DV, there is not much we can do about the carried-over DV artifacts, so we are forced to accept them, part of the codec.
Most of my work is for DVD delivery
The whole point of going to uncompressed from DV for me is to avoid the gamma shift that T-L> Compressor creates. The mpeg2 codec uses the field order of the source material, and I have not seen a field order problem with this method.
WRT scaling, Compressor’s presets for say mpeg2 for NTSC rescales to 720×480 size for 4:3 and Automatic size for 16:9, so the frame size of the un-compressed T-L is compensated for.

Cheers
Alexander

[Bill Lee]

[Alexander] “This test demonstrates exactly what we should NOT do, ie re-encoding to the same codec.

I’m arguing that re-encoding to another codec without understand what is happening to your video can be worse than re-encoding back to the same codec. In the case of using DV in the Apple Uncompressed Presets it can be demonstrably worse.

Why would we ever go back to DV?

Because of the KISS principle for most people. You are not ‘most people’, since you don’t need to go back to DV. Most people do (here I’m guessing), since they need to go back to DV tape at some stage, meaning that at some time they do have to run their video back into a DV encoder. If they don’t need to tap off a different output format before doing this, then they would be crazy not to do the whole thing in a DV sequence. Provided they follow some very simple steps, such as deleting any nested sequence render files and re-rendering the outermost sequence before final delivery, then they won’t run into more of the subtle problems caused by switching codecs, and yet can maintain the same quality gained by using an intermediate lossless codec.

In your post you mention that DV artifacts will not show when > un-compressed (they are hidden).

No, I said that DV artifacts will show when the DV NTSC clip is placed in an uncompressed timeline. You can’t get these artifacts out of the video, since they have now become part of the video. I’m sorry if I managed to give you that impression.

I assume we go un-compressed only when COMPLETELY finished (text aside), and then to delivery codec.

You can use an uncompressed sequence at any time, as long as you don’t have intermediate DV render files being used in this sequence. Just remember there are uncompressed sequences and Uncompressed Presets as supplied by Apple. They are not one and the same thing, since the Apple-supplied Uncompressed Presets are a subset of the huge set of possible uncompressed sequences.

DV is DV, there is not much we can do about the carried-over DV artifacts, so we are forced to accept them, part of the codec.

Very true, all we can try to to is to not make them worse by introducing more artifacts, such as would be caused by re-encoding them as DV. But sometimes we are forced to do at least one re-encode to DV in order to get our output format if this format is required to be DV.

Most of my work is for DVD delivery

Then dump your DV encoded clips into a lossless codec sequence before outputting it to your favourite compression program instead of taking it out as DV. What about video with each frame encoded as lossless PNG, or even the Animation codec? The down side of this may be the inability to play back in real time using that codec. You can use the Apple-supplied Uncompressed Preset for DV NTSC, as long as you remember to do two things: 1) Move the frame up/down one pixel to retain the correct field dominance, and 2) crop rather than scale from the 720 x 486 to a 720 x 480 frame size to avoid scaling issues.

Personally I feel that if you started with DV video, then DV and DVD are a reasonable match for quality and artifacts, and that it is usually not necessary to export as uncompressed as an intermediary step between DV and MPEG-2. Quality preservation from time spent on tweaking the MPEG-2 encoder to give better quality output far outweighs the quality loss from one more DV encode (IMHO). Subtle things like small amounts of black and white restore allow bits that might otherwise be wasted on detail in dark/light areas to be used to provide better detail in the midrange.

The whole point of going to uncompressed from DV for me is to avoid the gamma shift that T-L Compressor creates.

(https://docs.info.apple.com/article.html?artnum=304182)
See also Step 10) in (https://www.kenstone.net/fcp_homepage/basic_build_dvd_in_dvd_sp.html)
You can adjust the gamma as a filter in the Compressor presets.

The mpeg2 codec uses the field order of the source material, and I have not seen a field order problem with this method.
I believe that is because the MPEG-2 video on a DVD can have either upper or lower field dominance and will play back correctly. Compressor and then DVD Studio Pro might be sensing which field dominance is set in the video output from FCP and they both respect that setting and maintain it through to the final DVD (or maybe the default settings happened to be the right ones). As the Apple article on field dominance says, field dominance is how the data laid down on media is interpreted by the player as to which field should be shown first.

WRT scaling, Compressor’s presets for say mpeg2 for NTSC rescales to 720×480 size for 4:3 and Automatic size for 16:9, so the frame size of the un-compressed T-L is compensated for”

I believe this is where your video is being damaged, if it scales from (720 x 486) to (720 x 480) instead of cropping off the extra three scan lines top and bottom. Temporaly-separated scan lines are being interpolated and thus you will end up with a fields containing a mix of the other field.

Try it and see. Open a Photoshop file 720 x 486, with every second pixel row white and every other pixel row black. Now go to Image>Image Size and rescale it to 720 x 480. Do some white scan lines now appear where the black lines previously were? And vice versa? And do other scan lines now have a grey appearance, showing a mixing of two adjacent fields?

16:9 is just a flag to tell the player how to stretch the 720 x 480 frame.

In Compressor, you can crop the frame by those six pixels before you start the conversion, so you can work around the fact that D1 NTSC frame sizes are different to the DV NTSC frame sizes. It is unclear as to whether Compressor might be detecting the D1 frame size and automatically cropping it to 720 x 480 to make it suitable for DVD, so that you may not be seeing any scaling issues.

Bill Lee

[Rich Rubasch]

Absolutely excellent thread…

The scaling issue is one reason why we have become a DVCPro 50 house. We love it for our projects. Great keying etc. A lot of our stuff is shot on DVCAM. Not DVCPro50. The rest is Betacam. Everything comes in via the Aurora Pipe card SDI or Component to DVCPro50. Keying is great, color correction is great, file sizes are great…and when we are done and need to go out to DVD there are no scaling issues.

DVCPro50 might be a solution for this guy because at least it gets him to 4:2:2 color space.

Again, lots of great stuff in here. Still I am sticking to our DVCPro50 workflow as a perfect solution to our workflow.

Rich Rubasch
Tilt Media

[Alexander Kallas]

[billlee] ” It is unclear as to whether Compressor might be detecting the D1 frame size and automatically cropping it to 720 x 480 to make it suitable for DVD, so that you may not be seeing any scaling issues.
“

No, we’re talking about NTSC DV here, fixes at 720×480 frame size not 486. Compressor doesn’t know your final
format, just your choice in settings.
For DVD for television, there is no choice in Pixel Aspect Ratio for TV apart from NTSC-CCIR 601/DV
Compressor must be cropping automatically, I see NO scaling degradation.

Cheers
Alexander

[Raafi Rivero]

Here’s an add-on to the workflow question. I have mixed sources (DV, and stock footage that comes in at 720×486 often in photo-jpeg compression) and am aiming at an Uncompressed output that can later be laid to tape at a post house. My question is, should I downconvert everything I have to DV and output a 720×480 sequence that will later have to be scaled up to Uncompressed at the output stage, or scale all of my DV up to Uncompressed @ 720×486, and output Uncompressed from there?

Basically the question has to do with when will it be most advantageous for me to scale my DV-sized footage to the 486 vertical resolution.