Well, if you’re in a hurry, trying it in FCPX or Motion 5 may help. I hear many nice things about Adobe’s keyer as well, maybe try it in AfterEffects.
If you’re confined to working within FCP7, I’ll make a couple of suggestions, some of which may seem unorthodox.
In shooting, step one is to use the zebra bars function in your viewfinder as a final check for even screen lighting. Step two, for me, is to shoot a Macbeth chip chart under the actual set lighting conditions and to shoot a wide locked off “plate” shot of the green screen. Before I try to key the footage, I color correct it so the Macbeth chips are all where they should appear in my vectorscope. Now you KNOW what kind of signal the keyer is going to be getting. Just that one step has made any keying job I do 99 percent done on the first try. Amazingly, many newer users never bother with this prep step before asking a keyer to work with footage that may be too hot or too crushed or whatever.
Okay, you didn’t do those before you shot, but you can still make sure you’ve first color-corrected this footage, giving Primatte the widest range to work from.
Next; you can invoke multiple instances of the Primatte keyer and dedicate them to independently difficult areas. Take multiple samples with that eyedropper. I have taken bad greenscreen and first replaced it with more even, artificial bluescreen, then pulled the matte, and vice-versa.
Next, sometimes I cheat edges using the drop shadow tool: who says you must always make the shadow black? You can make the shadow chromakey green if you like, or chromakey blue, for that matter. Or the color of the object itself…
Another trick that works for many,(and the one most likely to help you now) is to take the raw footage into the 3-way color corrector and deliberately push extreme contrast and color changes that make the skin tones all wrong, in order to pull a clean matte, which you then add to a separate layer over your properly color-corrected footage as a traveling matte. A variant of this uses monochrome shots and luminance keying top cut a matte.
If all that still doesn’t help, you can brute-force it by exporting as frame movie, importing the frame movie into photoshop, and using the rotoscoping tools there (called the “extract” filter, AE also has this ) to hand-roto around each frame. If you’re handy with photoshop batch actions, this can be semi-automated and go pretty fast. Well, faster, anyway.
Good luck. Post revised results so we can all see what worked.