Green screen is an art in itself, and I would say most of the work is in the initial keying. If you don’t get that right, you’ll ultimately spend a lot of time adding effects on top of effects on top of effects… it’s not an ideal way to go. I’d recommend spending more time getting the keying right, and then everything else is more likely to fall into place.
One of the most tricky parts of green screening that I’ve found is when the green screen itself is poorly lit. This leads to lots of problems when trying to key it out. One method I have found works well for difficult shots is this:
1. Drop the footage into a new sequence that exactly matches the shot settings – resolution, frame rate, etc.
2. Add a Lumetri Color effect, and start tweaking the colors of the footage. What you’re aiming for is to get the green screen looking a much more consistent tone and hue. At this point, ignore what the subject looks like, it doesn’t matter if this looks off. Just focus on making the green screen much more even, without bleeding color into the subject. Also try and make sure none of the subject colors turn green in the process.
3. Sometimes it helps to turn the tint down slightly away from green, but use the saturation and hue curves to pull the greens back into a strong hue. If the green screen has a lot of shadows and highlights, you can isolate the greens and reduce the contrast using the HSL Secondary section.
4. Once you have prepared the shot so the green screen has flatter, more consistent colours, you can now drop the Keylight effect onto the shot (ensure it sits BELOW the lumetri effect or the color adjustments won’t change anything). Now when you eyedropper the green section, you should get a much better shot straight away. Play with the adjustments under the “Screen Matte” section a little, but don’t overdo it.
5. If you can’t easily get the Keylight effect to remove all of the green screen, then just focus on keying the darker green areas. Then stack a second Keylight effect on the clip and key out the light green areas. Much better to stack several keying effects on each other then to start messing around with rotoscoping etc.
6. If you can’t get a good key, sometimes you can go back and play with the lumetri colours again and try and enhance the greens a little more once you know what works and what doesn’t work.
7. Once you’ve got it keyed out, great! But the subject colors are probably looking wrong now. What I do is to use this keyed footage as a transparency mask – drop the same footage onto the timeline below the footage you’ve just keyed (or duplicate and delete out the effects).
8. Now you can use the track matte feature that is built in – you might have to “Toggle switches / modes” at the bottom of the Timeline panel to see this column. Then on the new footage, change the track matte to the the first footage where you keyed out the green screen.
Hope that helps! You’re on the right track doing this in After Effects too, this sort of thing is much harder to do in Premiere Pro.