I’ve had the same issue and ditching my Matrox RTX100 card cured it for me. Never looked back since. Slightly sorry to see the card go but it was the most unreliable piece of kit I’ve used in over 25-years. Would not capture for more than 20 seconds, created the effects you have witnessed and repeatedly locked-up.
What is your system setup?