In my case I had an effect on the video or photo layer and when I tried to mask out a section it left a thin dark line around parts of the mask. I fixed this by pre composing the video with all attributes included, then applied the mask to the pre-comp.
Hope that saves someone some time…