No matter what units you’re working with in your 3D app, (feet, inches, etc.) AE will always convert those numbers to pixels. So a 640×480 picture will initially look like it’s 640ft by 480ft if you tracked using ft. as your unit of choice. This makes for ridiculously large comps! I usually just animate in Maya and bring in the keyed data and camera info since it’s easier to animate 3D in maya than AE.
But changing your units may help by having to scale less.