There will be some clean up for sure but that’s par for the course.
If you can make a few different tracking point “designs” so that, as they appear and reappear there is some uniqueness the tracker can lock back on to. The differences can be as simple as color or shape of each dot.
Its also good to put the dots on key face geometry points so if you have to hand correct a few, you can line up the digital tracking marker to a specific facial detail. (a tear drop marker there the apex is on the corner of the eye for instance)
Remember you can also track backwards – sometimes this helps for markers that appear and then reappear.
Many times we will use multiple takes. One with just the face marked up, then one with just the hands with markers on them, then one with everything. You often get data from one take to work with another for a particular body part.