create a null(call it “guide”) and place it at your middle point, then link it to whichever tracked result you have that makes it stays there.
Then create another null and apply an expression to its position property.
a=thisComp.layer("guide");
a.toComp(a.anchorPoint)
This way you’ll get rid of any parenting rotation. Now you can link stuff to this null. If you still want scale, you should link it through expressions too.