jagged edges when using alpha matte

[Filip Vandueren]

Hi Wouter,

it’s a pain, but it actually makes sense: the edges are semitransparent, and in this way of compositing, you’re adding semitransparent white on top of semitransparent black at the edges, the result is not semu-transparant perfect white, but a grey that gives you his halo.

two possible solutions:
– precompose everything that’s inside of the mouth (BG-color, tongue, teeth) then apply the mouth shape masking. You could do that masking in a number of ways: outside of your precomp with a trackmatte, or inside of your precomp by putting it as the top layer transfer mode “Stencil Alpha”, or as the bottom layer, and have everything above it “preserve transparency”. (These two last methods will break if your precomp is set to collapse geometries though)

– Only works IF your layers are all shape layers. Keep everything as is while animating, but before you render: hide the Teeth layer, now add this effect to the bottom mouth: “Calculations”; second layer: teeth; second layer opacity: 100; preserve transparency: deselcted. Now add the effect “Set Matte”.

[Wouter Dijkstra]

Hi Filip!

I was already afraid this was just the insurmountable consequence of Alpha Mattes and using the original layer underneath it. Thank you for the solutions I will try them out! ☺

It’s a shame a function like Add Alpha doesn’t have any purpose here. (With mask + inverted mask Alpha Add works like a charm.)

It’s complex, but I’m happy that there is a way to solve it though. Great you were able to help me, it stuck to me 😉

[Oleg Pirogov]

Filip has summed it up.
My workaround would be to apply Curves effect, setting alpha channel to max:

[Oleg Pirogov]

Applying to the matte layer

[Filip Vandueren]

If you look at that against a non black background, haven’t your create actually jagged aliased lines at the black border with this method ?

[Wouter Dijkstra]

@Dave, You are right, the more you zoom the uglier it gets. But at 100% the border still shows.

The options Filip gave work quite nice. It just demands a little more work.

Thanks guys!

[Oleg Pirogov]

I did. You’re right, it’s not perfect. Though by stepping back from extreme alpha curve values, some relatively nice in-between result can be obtained: