I wouldn’t consider h.264 as an acquisition or editing codec, especially not for green screen work. It’s a delivery codec.
I’d acquire the footage uncompressed or maybe using ProRes 422, preferably 422HQ, if the clips are short and you have adequate storage. Good chromakeying follows the GIGO rule as far as quality of your raw shots.
I have followed this “light board” thread with interest for some time: I find it ambitious and intriguing, and doing it on a limited budget is quite the challenge. When it all comes together and the presenter is on their game, the look is distinctive and engaging.
I think from my reading of the thread I remember you have limitations on the markers and their colors, so the simple answer of “pick different markers” probably isn’t an option.
When building a chromakey, you are interactively tweaking a luminance key and a color-based key at the same time, and anything you do in color timing changes the key. I think a lot of people skip a step and don’t calibrate the raw footage *before* going into the keying of it. I try to always shoot a Macbeth chart on the set of the key, and I set up the initial color grade and highs/mids/blacks based on the chart. I find that when I take that step first, and only then start trying to chromakey, the “automatic” settings of most green screen fx plug-ins do a perfect or near-perfect job, needing only a very small tweak.
Real MacBeth charts are pricey. I lucked into mine. If you can’t swing buying one, find a color bar chart you can afford, or worst case, print one out in as high a quality as you can, on matte paper, and even if it’s not a great standard, it will at least be a *consistent* starting place for each shoot, under the actual existing lighting and lens. You can even make a chip chart out of formica or paint color samples from the Home Depot or whatever. The important thing is to have a consistent starting point and to get the levels right -before- you add the keys.
On the specific key you’re tweaking, where you’re so very close but not quite there… a couple things to try:
Lighting: can you add some more diffusion to the light panels? My keys were happiest with the light very flat and even, and the green at maybe 80 IRE, and well-saturated. Some people go for making the green too bright, and that’s counter-productive. Even and saturated is what you need, not blindingly hot. Small details might need a customized backlight on them.
Have you played with the “light wrap” settings in the keyer? Have you tried doing the key in a 2-step process, replacing the original captured green with a system-generated solid green, or blue, or some other oddball shade? Then take that new sub-composite and apply your keyer to that as if it was the original? I’ve done that, and where I had tricky edge problems I tried adding a hard drop shadow (verrrry narrow, like one or two pixels) that was eye-droppered to be the same shade as the green screen. When it went to the next step, I had less tearing-up of edges. This was one of my “secret weapons” on hard-to-key sources back in the late analog/early DV video days. The ease of adding drawn masks now to the shot, in multiple iterations, also makes keying jobs easier.