[Andras Peter]

Thanks Paul, I guess that would be one way of doing it.

I have realized that the best way to do it is reducing the opacity to zero for the keyframes where the tracking is not required. That does the trick.