Green Screen Dancing Edge Pixels (After Effects CS6)

[Steve Bentley]

Is it possible that as you work with premier (or whatever) and AE you are rendering your edit chunks out for use in AE and therefore the footage is being compressed and recompressed and recompressed? Compressing an H264 gets exponentially worse the lower the original bit rate is. And then if you recompress that, well, its starts getting very noisy. If you must render out clips to use in AE use a truly lossless format. Since it’s coming from H264 even Quicktime Animation best is lossless and high quality for that kind of process (and its runLengthEncoded so green screen stuff has very low file sizes without loss).
Are you color correcting the files in Premier before they go to AE? You want as raw an original as you can get for the key – every time to color correct you loose information. Try it – take an image in pshop and look at the histogram – now color correct that image, flatten the correction (make it so!) and then look at the histogram again. Instead of a continuous set of curves of the colors you get a spiky skyline – all those gaps are missing data.

H264 can work (as I think that clip I uploaded shows – at a higher bit rate at least) but the idea is to get as good a record as you can of the on-set stuff because something will cause you grief and the cleaner the matte you can pull without any tricks, the more tricks you have left to fix other stuff.
Did the AE file I sent with the key in it not work out well enough?
You could try another keyer – Primatte makes a good one – not sure if there’s a demo or the cost. There’s the granddaddy of them all – Ultimatte. Sometimes another compositor (Shake, Combustion, Nuke etc) will have their own keyer that will be dialed in just right to pick out details that others miss.
There used to be a pro version of the Keylight keyer too. It could tease out that much more of your edges.

In the end though, the clip you uploaded is really noisy so there will only be so much you can do without by-hand paint repair. When it comes to fixing green screen shots, rotoscoping certain areas is not a dirty word – most of the lord of rings got roto’d even with all their blue and green screen work. And don’t forget you don’t need a perfectly opaque matte – its ok to have a bit of the white part of the matte (the head) be grey – you can always make a garbage matte that is rough and fills most of that back in and leaves a feathered edge being a little transparent, and being able to see hair wisps at the edges is far better than helmet head.

[Steve Bentley]

Rereading your posts again: at that bit rate out of the camera you should be ten times better than what we saw. Grab a bit of the original camera footage and pull that into AE (ignore your edit list for moment – we just want to see if unadulterated camera originals can key ok.)
I’ll bet its the crush coming out of Premier that’s killing you.

[Mark Whitney]

[Dave LaRonde] ”
Understand that I have Not looked at everything in this thread.”

Hey Dave,
Ya the thread got a bit long but the issue turns out to be that he WASN’T using footage straight from the camera but that somewhere in his workflow was getting re-re-recompressed files instead it seems. That’s what he’s tracking down now. Needless to say, nothing is gonna help pulling a clean matte in that case.

[Matt Dodds]

I will comment on everything else when I get off of work, but since this forum can’t take over 100mb file size files, I had to shrink my footage to be able to be uploaded here. I took the footage into premiere and chopped the footage to 6 secs. When it comes to my work flow, I brought the footage from my SD card to a file folder then I uploaded it to AE. So the compression might be happening in the file folder or uploading it to AE. I didn’t put it in Premiere first, only to get it ready for upload on here. If you guys want, I can put it in my Dropbox, from my SD card, and share the link with you guys? That way, no compression from Premiere is happeneing.

[Matt Dodds]

Hey Steve, my work flow is SD card, to file folder, to AE. I don’t go to Premiere first, since I don’t need it in Premiere, until after I get it done in AE. Would moving it to a file folder be an issue as well? If you want, I can put it on my dropbox or google drive and share the link, that way you have it uncompressed. I only put it in Premiere, to sample a bit of my footage, since my file is over the upload file size on here!

I need to figure out what bit rate my camera does, but I looked at all the menus and nothing was showing me the bit rate. I don’t wanna go for Magic Lantern, since I don’t wanna break warranty!

I can’t get your file to work with my AE, since I have CS6 and cant work with AE CC 2018. If you can send me a CS6 format, that would help better!

I will look into all the keyers you mentioned. I am assuming they all cost. I had to buy Neat Video noise/grain reduction for my school projects, so maybe that might be something worth while!

I am not sure how to rotoscope, so I will research that. I just need to make sure I just have enough light and distance, so that way whatever technique I am able to use, it will make it way easier for me!

[Steve Bentley]

Sure upload a small chunk to drop box.
Just copying the file to a folder or uploading to the forum won’t compress it. You could just stick the original in AE and render out a small bit at full quality. If its 8 bit footage you could render out as Quicktime animation best or AVI uncompressed – both of these won’t introduce any compression (but the QT format will be smaller). If its 10bit or higher, set your AE project to 16bit and render out openEXR with floating point and RLE compression. This will render a bunch of stills.
You have to zip for drop box I think so just check your zip settings so that its just making a .zip wrapper and not actually compressing anything. The zip file should be just about the same size as the file you zipped. If its smaller you will know its been crushed. It may be a lossless crush but why take the chance when its the compression we’re investigating.
Once we’ve got the unadulterated footage i’ll send a CS6 file.

[Matt Dodds]

So what I did was I uploaded it straight to AE from my SD card. It is 8 bit footage, so I submitted it as a Quicktime animation. I didn’t zip it, as I can upload the clip straight to dropbox. Here is the link: https://www.dropbox.com/s/660rpfhuu1sz6eh/greenscreenfootage.mov?dl=0

I just wanted to make sure it had no compression. Thanks!

[Steve Bentley]

Ok that footage is miles cleaner than the one you uploaded before. However it’s still pretty noisy.

My preference for the color of green is more than just a personal preference.
In most CCD cameras (and ALL film cameras) the blue record is the noisiest, followed by the green and then the red (hence red-screen being the best, but unusable for humans – we once did a green-skinned alien movie and we used red screens – best keys ever!!!).
If you isolate each color in your footage to have a look (use the little kalidascope icon at the bottom of the main comp window and step through each color) you can see that red is pretty smooth. green isn’t too bad either. but blue is a noisy blocky mess!. If your green screen has more blue in it (ie a darker bluer green) it will be noisier. But if you go too far the other way into the yellow greens you are getting into skin tones and blonde hair colors.

Also – was the camera locked off? There seems to be some breathing going on with the frame. You can certainly do moving camera green screen shots but with your wrinkles you are creating noise that has to be figured out in the compression, and with the slight movement the “lava lamp” effect on the matte edge will be more prominent. BTW stabilizing the shot at this point will only make that worse.
Use the heaviest/beastiest tripod/dolly you can afford.

Let me have a play with this and we’ll see what works the best.

[Steve Bentley]

So here’s a comp with a few tricks in it (that are especially applicable to talking heads). Note that the project is in 16 bit even though you have 8 bit footage – this lets any color warps or level changes have a little more headroom instead of clipping.

In order to get rid of the blocky noise you have to crush the matte, this leads to helmet head. So we’ve added some hair back in at the edges. Its best to animate the head outline mask to follow the talent around (Rotoscoping). We haven’t done this but you can keyframe the mask as her head will undoubtedly bob along has she talks. In this case it doesn’t have to be a closed mask. We’re just making a glowing line where the hair is at the edge (and then hiding that glowing line) and that will act as a soft rim matte to let the edge hair through. You could also do this with inner and outer feather masks if you like, but this will have fewer mask nodes to animate.

We tried degraining the footage but grain isn’t really the problem – its the H264 blocks and general camera blobby noise that are the issue.

The main key is just a simple keylight key with some tweaking, I’ve also left on (but turned off) a few of the other native effects we tried to get rid of the noisy edge just so you might play and see what you get. We were getting better results handling the color and spill suppression on another layer unencumbered by the keyer. And the keyer controls, in this case, seemed a bit more of a big hammer than doing it another way. Just because you are keying a layer doesn’t mean you have to “see” that layer. If it gives you more options you can pass another version of that layer through the key – it’s just an alpha at that point. (you can even use a keyed layer as the alpha and then pass a copy of that layer with its own key through the original alpha – sort of applying they key twice.) Sometimes a more subtle, slightly transparent key, masked twice is better than crushing a single one too much. (but we didn’t do that here).

Ideally you should pull multiple keys on this shot for the various sections of the screen that are differently lit, and then mask off those areas and blend everything together. Then animate those masks to follow the action. Many many many green screen shots have a rotoscoped mask traveling along doing everything from filling the center of the actor in with good color corrected footage, to separating well keyed areas from areas of different color or density, to masks that remove tracking dots or wires or other appliances that were in the shot, or running a clean edge along scissoring legs that blob together as the apex closes at each stride. Unlike those behind the scenes DVD’s, it’s not just sampling the green with the eye dropper and you’re done.

Below the main key and its color element is another version of our edge hair trick – but this time with no mask. With it in behind the main alpha of the key, the wisps just poke out from behind the layer above. For both the hair edge layers: because the red record is so clean, if you shift that record to act as the alpha and then set all the others to red you get a cleaner less noisy element. (in this case we used luminance for each layer with red as the alpha just because we got a better signal to noise ratio)

Below that is the weird layer. The AltKey layer is almost the same as the top keyer, and again its turned off so we can pass another version of the main footage through. This time though we’ve blown up the main footage so that hair further in on the head (that is not interacting with the green screen) is passing through the edge of the matte. This fills in (even so subtly) hair at the edge and doesn’t have any of the green screen noise in it (or color) and again helps get rid of the harsh edge needed to clamp the noise away at the edge of the matte.

Finally on the background we’ve added some noise to distract from the very noisy front element. (we did our usual dirty quick talent collapse to get rid of her since we don’t have a clean plate)

There is one more problem you will have to animate the settings to deal with. It seems the talent is so close to the screen and the camera so close to the talent that when your crew moves about, the lighting on the screen changes. I can watch the matte get worse about 2 seconds before your clapper/loader comes into frame. In the two seconds before he appears, the screen gets darker and therefore the key changes and the settings may have to be keyframed to compensate. This darkening is him moving in for the kill and soaking up light as he comes – all things on set either reflect or soak up light. We also haven’t done any garbage mattes (for instance the bottom right corner where the key is much worse than the rest at these settings)
The final issue is a little moire pattern in her left collar (stage left). That is nothing to do with the green screen and is just an interference pattern between her blouse and the pixels in the camera.

There are about a thousand ways to do a screen shot. When we get a toughy, normally we pass it around the shop and everyone has a crack at it. We don’t share what we’ve each done with the next person so they take their own spin on it. Perhaps others will offer other solutions than we have done here (and this is by no means an extensive R&D project). In the end it has to be up to you because we won’t know whats important to you. You may be fine with a clamped edge giving her a crisp outline as long as it gets rid of the noisy edge. You may prefer a bit of noise if it gives her a softer edge and some wisps. You may prefer a blurred edge to give it a “look” and thus remove the noise altogether. And every set up you have in your shots may require a different method due to lighting, colors, talent or movement.

12293_greenscreenheadcomp2.aep.zip

[Matt Dodds]

I have a Manfrotto MT055XPRO3. It was locked off and it was not moving. Once I started recording and used the clapboard to sync sound, I left it allow until I was done recording!