[Alex Spoettel]

A very effective inbuilt method is using “Effects>Matte>Refine Hard Matte”. This works really well for jagged edges / edges with artifacts on alpha mattes – which is exactly what you need to fix up the Wave Warp edges.

It has many customization options and advanced parameters like feathering etc. that help smooth out the result and make for a great procedural solution for this problem.

Only downside: it is a quite costly effect depending on the parameters used so it is best only enabled when rendering the final comp. A way to speed up rendering is to set “Reduce Chatter” tp 0% as this is irrelevant for jagged edges and is very hardware-intensive.