They are never pixel perfect. You always have to expand by a pixel or so. The part of your problem I couldn’t comment on is why you needed an expansion of seven pixels… seems like a lot.
Here is Why:
They are not pixel perfect because when you draw a mask it is a vector shape. The image being masked is comprised of pixels. No matter how far you zoom in your pixels will degrade but the mask will remain straight edged.
If you are using a circle mask it has to average the pixels it crosses when it renders, be it a RAM render, or a queued render.
Go into your same project… but make your mask a square rather than ellipse and I bet you a hundred bucks the edge difference goes away between the add and subtract layers. Try it.
I hope that makes sense.
My Work:
https://www.morecompletefx.com