Dealt with the exact same problem. I know how to fix when I track in Boujou and use NUKE but After Effects posed issues for me since it’s really still 2.5D. It was a bit difficult to wrap my head around. Not sure if you’re still having issues or are looking for a permanent solution but this literally solved my problems with this with a couple of clicks and it’s only $19.
https://aescripts.com/orient-world/