One thing you can try is a “light wrap” effect. It’s usually used when you have an actor or object composited over a background and you use the edge of the actor’s matte to create the effect of light wrapping around the actor as it would in nature. Here’s how. Create a matte from the actor if you dont have it already. Take the matte and create a comp in AE. Duplicate the actor matte layer. On the top layer set the transfer mode to “silhouette luma”. This will create a white outline of your actor. It will be a little rough so you’ll want to make some adjustments. Still on the top layer, add a “levels” effect. Adjust the slider to bring up the white level in the outline of the actor. After you bring up the levels you want to apply a blur to the same layer. The amount of the levels and blur will depend on how much of the “light wrap” you want to show, but you can adjust that later. Next, create a new comp. You will have your BG layer(on the bottom), then the actor(above BG), Then another copy of the BG layer above the actor, then your final top layer will be the “light wrap” or outline layer(comp) you created earlier.Next, you’ll set your uppermost BG layer’s(the one above the actor) transfer mode to “Luma Matte” of the above “light wrap” matte(outline). What this is doing is using the thin outline as a matte for the BG layer and setting it on top of the actor. The effect is subtle but effective. If you want to see more of the effect go back to your original “light wrap” comp and up the levels and put more blur on the outline, which creates a bigger matte. Play around with it.If you want it to be seen less just bring down the opacity of the BG layer. This is just one way to help your edges, hope it helps. GOOD LUCK!!