Thanks for your questions. I was connected with firewire 6 to 4 pin and monitoring through the headphone jack on the deck.
I finally figured out the problem with the help of someone at apple. It was apparently interference somehow from the firewire cable going to the external hard drives. I shut everything down, turned everything off, the hard drives too, unplugged the hard drives, restarted the computer, then tried opening a final cut project using a quicktime file on my laptop and the output to the VTR worked fine. Then I reconnected the hard drives and everything is working fine with them too — for now.
The apple guy said this happens sometimes, the hard drives give some kind of interference going through the other firewire port. He recommended using eSATA to avoid this.