Increasing mask expansion is not the same thing scaling a mask. Scaling preserves shape. Expansion says “Make it bigger by growing outward from each point a certain number of pixels.”
For each point on the original mask path, imagine a circle radiating outward by the number of pixels by which you’re expanding.
I’ve embedded an image below that should show what I’m talking about. The white rectangle is the original mask path. The red circles indicate the expansion radius. Note how these circles define a sharp inner rectangle, which is what you get with a negative expansion; and they define a rounded outer rectangle, which is what you get with a positive expansion.

———————————————————————————————————
Todd Kopriva, Adobe Systems Incorporated
putting the ‘T’ back in ‘RTFM’ : After Effects Help on the Web
———————————————————————————————————