My understanding of how the two apps work to geth is use FCP to to your video and once you have finished makiing the video look perfict you send it all to sound track and make all your audio adjustments there. Shure you can do basic sound in FCP but Soundtrack gives you much more control.
What are yoou capturinig from (format device etc) and what are your settings at in FCP use the easy set up and choose the proper format that you are trying to capture.
It all starts with geting the best quality video footage in the first place, then encoding. Remember that most commercial DVD are also Dual-Layer so les compression is required.
First look at the footage played from the tape that would tell you if it was the encoding that made it look so bad.
Another eazyer way to pur your video strate to DVD is to siplpy hook the camera up to a DVD Recorder they start at about 100 and many have a wirewire jack to hook strate to the camera