Here are some more images to show what’s going on.
This is the original footage after I’ve used Primatte to key out the background.

I duplicated the layer and set the blend mode to Add in order to brighten it up. You can see how the foreground object looks bright, and you can see the edges of the key. (you can also see my mask outline, but it’s set to None.)

I created the mask and set it to subtract, and feathered the edges. Now it looks great. The foreground object is dark as it should be, and the key edges are gone.

But when I render, it looks like the middle image – like my subtraction mask is gone.