Here’s what they’re teaching these days in the Advanced FX class:
1. Drop a 3d warp on the clip you need to chroma key.
2. Enter effect mode, and select Chroma Key from the Foreground parameter.
3. Use the eyedropper to select the primary key color.
4. Use H Tol to adjust the range of colors to be keyed out.
5. Use S Low to close any holes in the key (easier to see if Show Alpha is activated).
6. Under the Erode/Expand parameter, adjust Filter to control the pixels at the edge of the key and smooth out aliased edges.
7. Use Center to set the amount of erosion or expansion. Values higher than 127 choke/erode the matte.
8. Use Soft to blur the matte edge.
9. Under the Spill Supression parameter, activate Use Key Color (the color in the color picker will not look like the key color; this is ok).
10. Increase Gain until you can no longer see the key color in the foreground image.
11. Use Soft to blend the replacement color into the surrounding colors.
So for your edge problem, I’d try the Filter and Center parameters under Erode/Expand to remove the haloing of the key color around the subject.
good luck,
Carl