There are two things I can think of that can be causing the distortion you are hearing. The first is that the line level being sent from the mixer is overloading the input preamp of your camera. Adjusting the level controls on your camera won’t fix this. You need to lower the level coming out of the mixer. You can do this by adjusting the “Sends” on the mixer. If this is not possible, you need to use a “pad” that will attenuate the level. A pad looks like a tube with a connector on each end. You plug one cable from the mixer output into one side of the pad and a cable to the camera into the other side of the pad. You’ll need two pads as they are typically mono devices. Usually pads are switchable so you can pick an amount of attenuation that eliminates the distorsion, but doesn’t make the level too low, ie. -10 or -20db. The second thing you need to check is that the cables running from the mixer to camera are wired correctly. You mentioned RCA connectors. Are the outs on the mixer and the inputs on the camera both RCA? If so, then the problem is most likely the output level being too high. If one end is RCA (2 connections: center and shield) and the other is XLR (three connections: high, low and shield) you must make sure the cable is wired correctly. If the cable’s not wire correctly you’d hear tinny and/or hum-filled audio, probably not distorsion. Hope this helps.
Dual 2.5 G5 4 gigs RAM OS 10.3.8 QT6.5.2
Cinema 23 Radeon 9800
FCP4.5 DVDSP 3.0.2
Huge U-320R 1TB Raid 3 firmware ENG12.BIN
ATTO UL4D driver 3.20
AJA IO driver 1.3.1 firmware v21-26