Out of Range Rendering Failure

[Kevin Cannon]

How is your AJA HI5 setup? It there a color range (0-255/16-235/0-1023 etc.) setting like the HDP2? I’m not totally sure what you mean by “scaled into legal range but stored outside of legal range.” How are you checking your renders?

I get practically identical results from these three:

DaVinci “legal video” >SDI> HDP2 set to 16-235 >HDMI> Dreamcolor calibrated for rec.709 (black level 126)
DaVinci “legal video” >SDI> FSI Monitor in rec.709
DaVinci “legal video” >HDMI> Panasonic Plasma calibrated to rec.709

I would imagine that if the Davinci was set to “unscaled full range” and the HDP2 (or HI5?) was set to 16-235, that the Davinci would retain a bunch of information falling between 0 and 64 while the Dreamcolor would not display any of that information under it’s (incorrect) black point of 64. Then it would make sense that any render viewed in a way that didn’t go through the HDP2 would have more detail (from 0-64) or washed out blacks (at 64), and it would make sense that switching to “legal video” would suddenly reveal all that detail between 0 and 64 that just got scaled to be 64 and above, so suddenly visible to the Dreamcolor.

I dunno though, just a guess.

KC

prehistoricdigital.com
hardworkingpixels.com

[Darin Wooldridge]

This is for scaling from 0-1023 to 64-940. not to legalize.
It never affects the render just the video output for layoffs to tape.

Not a good idea to change mid project.

NOTE: The comments above are strictly mine, and may not necessarily
represent those of my employers.

Darin Wooldridge
Colorist / Technical Strategist
818-653-3918-cell
dwooldridge@mac.com
check me out at https://www.facebook.com/pages/Davinci-Resolve-Colorist/117363011609028?ref=….

[Bill Nelson]

Kevin’s on to something, I think. Any other general output hints out there? Sam, what are your destination formats/masters?

[Samuel Bilodeau]

That is exactly what I first thought. However, when I add a LUT to clip everything 63 and under (and 960 and over), the image doesn’t change in either of the viewing modes,(though I see the clip on my WFM). When the viewing mode is unscaled, I’m seeing the crushing of black at about 127, but when it is normal scaled, I’m seeing the crushing at 63, as I would expect.

Since our normal DaVinci colorist is out of town to recreate the unscaled look, in the mean time of waiting for the creation of a LUT to do the translation, I have done a post-render crush of the blacks and whites to rescale out of the more limited scaled-in image, restoring the observed image from the ‘unscaled’ viewing mode by matching points on the observed waveforms (more on that later).

The practical result is to most displays and people an imperceptible loss in potential luma dynamic range, of about 13% (rather than rendering out 1024 values, I’m rendering out 898 luma values and scaling them back into 1024); since ultimately most people will see an 8bit DVD or Blu-ray, with a 75% loss of luma dynamic range (before MPEG-based compression), the client felt the loss (which he could not perceive) was acceptable.

I will keep looking into the issue for now – I did suspect that there is something wrong in the interplay between the HI5-3G and the Dreamcolor with the color handling resulting in this high clipping when I feed an unscaled image through. However, when I bypass the HI5-3G and feed my Videotek VTM directly off the Decklink, I see the same result on the image coming out of the Videotek, in that when the image is unscaled in DaVinci, the blacks seem to clip at about 127 instead of 63. However, my onscreen WFM assures me that I am getting signal with values below 128, which puzzles me.

If I can isolate the root problem in the video pipeline, I’ll post back an update. For now I am able to create a ProRes 4444 master that renders out the right look at the proper gamma.

Samuel

[Kevin Cannon]

So I’m still going with my theory here – as Darin points out, the setting only applies to the monitoring, not the actual image, and the HI5 settings are the major adjustable part between the Davinci setting and the Dreamcolor… did you connect it via USB in AJA miniconfig to check? Especially if RGB Range was set to auto – then it could potentially be changing and you’re trying to hit a moving target. I’d be curious to see what they are set to. Great that you were able to work adjust the final image post-render into how it appeared on the monitor…

when I add a LUT to clip everything 63 and under (and 960 and over), the image doesn’t change in either of the viewing modes (though I see the clip on my WFM).

If information can be clipped to 64-940 (seen on the WFM) without making a difference on the Dreamcolor, that would suggest the Dreamcolor wasn’t displaying full range 0-1023.

When the viewing mode is unscaled, I’m seeing the crushing of black at about 127, but when it is normal scaled, I’m seeing the crushing at 63, as I would expect.

Seems like something is doubling up here…

“when I bypass the HI5-3G and feed my Videotek VTM directly off the Decklink”

Were the scopes always being fed from the HI5? That seems like a potential complication.

“when the image is unscaled in DaVinci, the blacks seem to clip at about 127 instead of 63. However, my onscreen WFM assures me that I am getting signal with values below 128, which puzzles me.”

ha me too. But I’m not very familiar with the videotek vtm… if the blacks are clipping at 128 on the hardware scopes, sounds like something is doubling up…

KC

prehistoricdigital.com
hardworkingpixels.com

[Samuel Bilodeau]

“Were the scopes always being fed from the HI5? That seems like a potential complication.”

No, not even close. This is what was boggling me – I removed the Dreamcolor and the AJA box from the downstream video pipeline and attached my scopes directly to the Blackmagic Decklink’s output coming out of the computer.

“ha me too. But I’m not very familiar with the videotek vtm… if the blacks are clipping at 128 on the hardware scopes, sounds like something is doubling up…”

If it were a strict scaling issue I would expect to see a problem when I turned on scaling for DaVinci’s output, instead of when I turn it off, or at least see a difference in some image when I clip off the under 63 values, which I don’t see.

Anyway, after giving it a day’s thought, I ran some tests and seem to have figured out exactly what is going on with those settings in DaVinci.

Since my Dreamcolor and my WFM were both showing the same shifting between the two settings, with the WFM being directly attached to the Decklink, I assumed that the issue was not with the downstream video settings – both the Dreamcolor and the WFM were behaving as expected with SMPTE standard devices.

Next, I eliminated the Decklink as the problem, by switching to multiple applications, all of which produced images equal in WFM and appearance (I used the same clip) to the image produced by DaVinci under the settings “Normally scaled legal video”. Here, my scopes let me see both the SMPTE black point and the unclipped data that existed outside of legal range but was being clipped by the displays, as was to be expected. The scopes also matched the values I saw for the clip in the DaVinci internal WFM. The Decklink therefore is passing the video signal through to my display pipeline unaltered.

So, if the card and the downstream video were doing exactly what I expected, it seemed there must then be an issue with DaVinci.

What I predicted based on my observations was the following: The “Unscaled full range data” was in fact taking the video data that fell into legal range (64-940) and scaling it out of legal range (back into 0-1023).

This, I predicted, would cause the observed white and black point shifts that I had measured on my WFM. The Blackmagic would pass these scaled values to the SMPTE displays, which would then clip the white and black point again.

To test this hypothesis, I mathematically calculated the effect on the black and white points for the original video output, based on the scalar range, and came up with the following values: White would be 868, Black would 118 (See the math below). So, I created a soft LUT that clipped the output video at these point values.

I applied the workaround to recreate what I had observed, passing it this clipped video. I found that it appeared identical (on both the WFM and the display) to the same clip with no clipping of the values below 869 and 118, when the workaround look was added to the output clip, excepting that the clipped video raised the black point to legal level, visible on the WFM.

So in essence, what I figure is that when I use the “Unscaled full range data” selection, what I am actually seeing is legal video scaled into unscaled ranges, and that is properly clipped downstream by my SMPTE displays and waveforms, resulting in an improperly clipped black and white image of higher contrast.

Mathematical Justification of Black and White Points:

Ratio of Legal range to Full range= (63:940):(0:1023) = 876:1024 = 1.166
Observed Black Point = Number of SMPTE Legal Black Clipped Values/Ratio + SMPTE Legal Black = 64/1.166 + 63 = 118
Observed white Point = SMPTE Legal White – Number of SMPTE Legal White Clipped Values/Ratio = 940 – 84/1.166 = 868

Sam