Gamut errors are USUALLY, with minor exceptions, generated when the Luma of a particular video image goes over 100 IRE and those offending parts have no color representation (in other words, the info that generates color in those highlights is missing.
You may say, but wait, I know that video can actually be recorded on tape to 120IRE Paul. How come over 100 IRE Luma signals cause Gamut errors?
Let me put it this way… some equipment manufacturers are not too cool, and make equipment designed to only accept Luma signals between 0 and 100 IRE. So, while your monitor and video tape recorder may work with 120 IRE of Luma, the TV transmitter/cable headend will spew it out. YUK. Legally speaking, NTSC accepts Luma up to 100 IRE, and therefore, transmitting Luma over 100 IRE is grounds for loosing your over the air license in the USA.
When a Gamut Error is transmitted via an NTSC transmitter, the parts of Luma that are over 100 IRE will appear digital dark green. You may notice this ugliness in live transmissions where Sunlight unexpectedly creeps in the shot, the whole video interconnection is digital, and the resulting Luma level jumps to 150IRE. (Normally in outdoor events the Luma is clamped to 100IRE for these reasons.)
To fix a Gamut Error issue:
1) Artificially clamp all your video Luma levels to 95 IRE.
2) Lower your Luma levels to 95 IRE.
If you have not edited on linear based digital video systems before, these two suggestions may not mean much to you. So, think of this as reducing the white level. In 8-bit YUV video, 100IRE is the same as R=235, G=235, and B=235, so you lower these signals to say 223. In 10-bit video, 100IRE YUV is the same as R=940, G=940, and B=940, so lower it to 893. This would theoretically get you Luma levels that peak at 95 IRE.
When you only work in RGB color space, it is technically rare or impossible to go over 100 IRE. HOWEVER, if the RGB video is changed to YUV color space, it is very easy to get video with Luma levels over 100 IRE.
Notice I said that YUV reaches 100IRE when the RGB values are 235 in 8-bit video. If you raise the RGB color to 236 or higher, say RGB 255, you will imediately get Gamut Errors over those parts that are over 235. Remember, this is only in YUV color space video, not RGB color space video.
In RGB color space, 8-bit video can actually reach 255 for the RGB intensities, and you will be peaking 100 IRE. In 10-bit video (RGB color space), 1023 is the same as 100 IRE.
All digital video tape (with the exception of HDCAM SR) records in YUV color space.
To fix your problem, you may want to lower the white component of the RGB signal in those offending frames and you should be ok.
-Paul