Every greenscreen key is a little different, so there is no one answer, but I’ll throw a couple tips your way.
1. sometimes adding a “levels” effect prior to keylight helps. You’ll have to play with it and even go into the separate RGB channels and adjust. You’re trying to even out the green and make it true green, use the info tab to monitor.
2. When you choose the screen color you want to key out, place the eyedropper in a green area nearest the area you are having difficulty with. So if a hand is getting cut off, try selecting the green that is near that hand.
3. Move your cursor to a frame that has the worst key and change view to “screen matte”. Now adjust the settings for “screen gain”, “screen balance” and then in Screen Matte play with “Clip Black” and “Clip White”. You’re trying to get the image to be pure black and white with no noise of either. Then switch back to Final Result. Screen shrink and softness work together. Sometimes going -1 on screen shrink and then 1 on softness works well. Also, try changing the replace method from soft color to hard color.
4. You also might try using the AE matte choker in tandem with Keylight. You will probably have to dial back keylight and then use matte choker to clean the edge.
It involves a lot of trial and error, so it’s hard to help you out without seeing the footage. Just keep tweeking it. I don’t have any advice on how to shoot it as I just do the keying.
Good luck,
Kevin
System Setup (for a more detailed list, see my profile)
HARDWARE
Mac Pro 2 x 3 GHz Quad-Core Intel Xeon
9GB Memory
Two 16x SuperDrives
Dual-channel 2Gb Fibre Channel PCI Express card
Apple Cinema HD Display (23″ flat panel)
ATI Radeon X1900 XT Graphics Card
AJA Kona LHe SD/HD capture card
Apple Xserve Raid 5.6TB
SOFTWARE
Mac OS X 10.4.11
Final Cut Pro Studio 2
After Effects CS3
Photoshop CS3
Illustrator CS3
Boris Continuum 5
Sapphire Plug Ins
Roxio Toast
Digital Anarchy Anarchist Suite
ParticleIllusion 3.0
Trapcode
Zaxwerks