With CS6, you can now delete points but this has to be done for each frame that the point exists. In AE CC, you will be able to delete points across all frames. The idea here is to obtain a more accurate camera solve.
What you can do is to mask out irrelevant areas in a precomp. Then hope that you get more points in the areas where you need them. I should add that this has a low probability of success but it’s worth a shot.
I’m also surprised that a single point will not do the trick for you. The only difference between a single and three points (heck, you can even select as many as you want) is that you will not have an accurate orientation value with a single point solve. You can however nudge the orientation values to suit your exacting needs.
If all of these fail, you will do well to post your video so that we can have a more direct appreciation of your situation.
HTH
RoRK
Intensive mocha & AE Training in Singapore and Other Dangerous Locations
Imagineer Systems (mocha) Certified Instructor
& Adobe After Effects CS6 ACE