[Jean Drand]

My post is not a precise answer to your question but it can help you to understand why there are clipping issues in After Effects an how to solve the problem. I have just write this post to another user wich has a similar problem. I use 8bit for the explanations but the same is true for the other bit deephts

Excuse my bad English

This issue is very common. Due to the color difference encoding math, Y Cb Cr (and all the YUV encoding processes) can encode color levels that are fare above the RGB color space. Almost all the camcorders record Y in the range 16-255. For ex.: If Y=255 and Cb and Cr=128, this is equivalent to RGB 255,255,255 but if Cr and or Cb are not 128, this codes for a color that has the max RGB luminance but is not white and can thus not be converted to RGB because it is a color above 255. Even if you limit Y to 235 when shooting, a big lot of colors can be above 255. The fact that the 16-235 YUV range is further scaled to RGB 0-255 exacerbates the clipping problem.
For ex.: my Sony has shoot Rec.709 YCbCr=244, 63, 207 (bright red part if a lava flow) = RGB 365.63!!, 219.75, 126.08. I have very often G or B=290. This is a conversion to RGB assuming that my YUV is 16-255 for Y, but if I assume it is 16-235, then the resulting RGB values are scaled a lot higher due to the conversion from 16-235 to RGB 0-255. In Rec.709 YUV, the max. encodable unscaled values are about RGB 400, but only 2 colors can be above 255 at the same time. The max encodable color level for a pixel depends on the levels of the other colors of this pixel.
If your TV or Pro monitor is XVycc (=x.v.Color) compliant (case by Sony, Panasonic and Philips), those out of RGB space levels are correctly displayed without any clipping when you connect your camcorder HDMI on the monitor. This TV’s and monitors convert internally 8 bit YUV to 9 bit RGB, wich can hold RGB levels from 0 to 512. At the moment, the only solution to see the whole signal on a suitable external YUV monitor when you grade is to use Edius with an Grass Valley output card or with the very good Blackmagis Intensity Pro 4K. Edius import your YUV clip as is, send the datas as is to the external monitor and also to the output YUV codec without clipping!
The managment of the data levels in the Adobe programs is catastrophic. Never import YUV because it is clipped at 16-235 at import time and this 16-235 range in then scaled to 0-255. In Edius or in Resolve, lower the gain to have all the color peaks under 255 (=1023 in Resolve) and export in 16 bit RGB (for ex. in TIF, Edius can only export 8bit Tif’s ). This creates a darker RGB file but with unclipeed colors. Import this in Adobe and export from Adobe in 16 bit PSD or TIFF (in fact, it’s only 15 bit because the Adobe range is 0-32767 and not 0-65534). Now, you can reimport the clip in Edius or Resovle and gain it back up to his original level. Be aware that by doing so, the black level is not correct in Adobe because it is between 0 an 16 instead of 0, but this has no importance if the purpose is to only apply moving effects. Be also aware that Resolve import YUV as is but, even if properly configured, send only the at 255 clpped levels to the external monitor an to the output codec, wich is not the case with Edius. Be also aware that the conversion between YUV and RGB is highly destructive and introduces color banding. If you must absolutely do this conversion, use always a codec that has an higher bit deepth than your source and use the gain reduction method described above. The whole RGB range falls in the YUV space but not the opposite. In Resolve, you can render in 2 16 bit RGB float math codecs wich can hol the full YUV color range, but the levels are miss interpeded by Adobe. If the purpose is to make full camcorder quality masters, you must chose an NLE that has an internal YUV core and that does no any level conversion of the souce clips. Many many NLE’s don’t satisjy the ITU-BT specs wich says: “The original camcorder data levels that are under 0 or above 1 may never be clipped during production nor in the intermediate files. In a pro editing application, the transmission of all the video information (including ivalid and illegal levels) between equipment is required. Masters must always contain all the original video levels that can be encoded in the chosen Rec.”
I am waiting for your reaction.

[Jean Drand]

Excuse my bad English

Your issue is very common. Due to the color difference encoding math, Y Cb Cr (and all the YUV encoding processes) can encode color levels that are fare above the RGB color space. Almost all the camcorders record Y in the range 16-255. For ex.: If Y=255 and Cb and Cr=128, this is equivalent to RGB 255,255,255 but if Cr and or Cb are not 128, this codes for a color that has the max RGB luminance but is not white and can thus not be converted to RGB because it is a color above 255. Even if you limit Y to 235, a big lot of colors can be above 255.
For ex.: my Sony has shoot Rec.709 YCbCr=244, 63, 207 (bright red part if a lava flow) = RGB 365.63!!, 219.75, 126.08. I have very often G or B=290. This is a conversion to RGB assuming that my YUV is 16-255 for Y, but if I assume it is 16-235, then the resulting RGB values are scaled a lot higher due to the conversion from 16-235 to RGB 0-255. In Rec.709 YUV, the max. encodable values are about RGB 400, but only 2 colors can be above 255 at the same time. The max color encodable level for a pixel depends on the levels of the other colors of this pixel.
If your TV or Pro monitor is XVycc (=x.v.Color) compliant (case with Sony, Panasonic ans Philips), those out of RGB space levels are correctly displayed without any clipping when you connect your camcorder HDMI on the monitor. At the moment, the only solution to see the whole signal on your external YUV monitor when you grade is to use Edius with an Grass Valley output card or with the very good Blackmagis Intensity Pro 4K. Edius import your YUV clip as is, send the datas as is to the external monitor and also to the output YUV codec without clipping!
The managment of the data levels in the Adobe programs is catastrophic. Never import YUV because it is clipped at 16-235 at import time. In Edius or in Resolve, lower the gain to have all the color peaks under 255 (=1023 in Resolve) and export in 16 bit RGB (for ex. in TIF, Edius can only export 8bit Tif’s ). This creates a darker RGB file but with unclipeed colors. Import this in Adobe and export from Adobe in 16 bit PSD or TIFF (in fact, it’s only 15 bit because the Adobe range is 0-32767 and not 0-65534). Now, you can reimport the clip in Edius or Resovle and gain it back up to his original level. Be aware that by doing so, the black level is not correct in Adobe because it is between 0 an 16 instead of 0, but this has no importance if the purpose is to only apply effects. Be also aware that Resolve import YUV as is but clips above 255 when you render, wich is not the case with Edius.
I am waiting for your reaction.