Just realised what the problem is.
The solid layer is much bigger than the comp and video footage layer.
If the solid is the same size then everything lines up perfectly.
Trouble is I need the solid layer bigger in order for it to track the features on the video without running off the end of the solid layer.
Not sure what the best solution is here.
Edit: Simple solution! Precomp the solid layer and then pan/scale/rotate the precomp as needed.