It is not clear, in your post, if you adjusted points on the tracking spline or the Planar Surface. Take note that you should be adjusting the points on the tracking spline since the Planar Surface is essentially a child layer of the tracking spline. IOW, the Planar Surface reacts to tracking data that is acquired within the tracking splines.
For tracks that seem to misbehave or require finessing, you should use the Adjust Track module. It is in this module that you manually adjust the Planar Surface to perfectly fit your tracking requirements if the tracking module doesn’t provide you with sufficiently precise tracking results.
With specifics to your tracking task, I would try to track the entire screen instead of the parts that you have indicated with the tracking spline. A larger area will almost always lead to more tracking accuracy. You should also look at applying Perspective tracking and ensuring that Percentage of Pixels used is set to a high value – 90% will be a good number.
HTH
RoRK
Intensive mocha & AE Training in Singapore and Other Dangerous Locations
Imagineer Systems (mocha) Certified Instructor
& Adobe After Effects CS6 ACE/ACI