That wobbly surface tension edge of the frame looks to me to be overly aggressive keylight matte cleanup. The shrink/expand matte options in keylight often do this sort of thing. Best to try to get the best key you can before using those cleanup options. You can try adjusting the color that you are keying. Sometimes it works better to just move it manually around the color picker and try to find a color that gives you the best key rather than try to sample it. For poor greenscreens, I sometimes ditch keylight altogether. I personally hate the rotobrush. I find it takes forever, and it often has bad chatter. I prefer traditional rotoscoping over using it. If you can find a setting that will give you a good edge, but has holes in the figure, you could animate a garbage matte to fill those holes.
Here are alternative tricks I’ve come up with for pulling tough keys. Sometimes they work.
1: Try Premiere. I have enormous respect for Premiere’s Ultra Key. It often pulls miraculous keys that I wouldn’t think possible. Doesn’t always work. Keylight gives you more options, but some footage keys better in keylight, some in ultra key. I wish the Adobe team would include Ultra Key in AE as an alternative keyer.
2: Try the paint bucket effect. I have sometimes had good results with the paint bucket effect applied to an unused corner of the badly lit greenscreen. Sometimes you need several instances of the effect in different corners. When this works, its fantastic. You can even do everything with one effect, check Invert Fill, and set the blending mode to stencil alpha. This trick works well for people shot on black or white which don’t key out well with keylight which assumes a colored screen.
3: Try multiple instances of keylight sampling different greens from different sections of the poorly lit screen. Use less aggressive matte cleanup on your keylight, and set the view setting to “intermediate” on early instances to avoid color correcting the greens that you are wanting to key out.
4: If the camera is locked, and you have a clean frame with the actor not in the shot, (or if you can make such a frame). Sometimes a difference matte works.
5: I have sometimes gotten good keys by duplicating the layer, and applying shift channels to the top layer. I set all the channels to full on, then cycle the alpha channel through the different channels till I find the one with the most contrast. I then use the levels effect (on the alpha channel) to increase that contrast. I then use that as a track matte for my color layer.
Just some things to try. I hope one of them helps.
-Matt
Need a quick break from motion graphics?
Try my game Constellation at:
https://www.paperdragongames.com