Hey there thanks for the suggestions.
My original approach was 2D motion tracking, but the map seemed to confuse AE a lot.
I have tried applying a 3D camera tracker – only problem is that it attaches my stripped background to the cameras postition (the z-position).
Is there a way to make it respect the z-pos coords but make sure they are behind the map it self – and that they dont attach to the camera?