Mike,
I had a simular problem previously. After much searching on this site and Apple’s forums I found out that you can import PSD menus at a higher resolution than I first thought – 300dpi.
Also what really helped was bringing the menu in as 720×576 and then setting the menu to 16×9 once imported into DVD Studio Pro. This meant I created the menu as a 1024×576 graphic, then resized it to 720×576 so it appears anamorphic ready for importing into DVD Studio Pro.
One other small trick is to add a 1 – 3 pixel ‘blur’ in photoshop.
Regards
Bridget