Sounds like you have an impedance mismatch. You should make sure that the camera is set to line in and the mixer is set to line out. If you send line to mic level inputs you will get distortion not matter how low you set the record levels on the camera.