Thank you all for the suggestion to use mocha for tracking. Just used the 2D tracker in mocha but with the perspective option turned on. Took a little bit of tweaking but now the shot looks believable.
For what its worth I did try the AE 3d tracker and it wasn’t doing the job either, but its quite possible I just wasn’t doing it right.