Encore’s menu editing system is based on Photoshop files. Everything is based on layers in a PSD files. Your button elements need to be turned into layers.
You don’t necessarily need to cut out the button areas from your background image, but you at least need to create a layer set for each of your buttons that contains the button region.
If I’m understanding what you have already (and I’ve never really used DVDSP), here’s what you want to do:
1. Open your background image in Photoshop.
2. Add each of your button overlays as layers.
3. Get rid of the empty space in the overlay layers.
4. Put each of the overlay layers in it’s own layer set. Make sure the layer set name starts with “(+)”, which is what tells Encore that the layer set defines a button region.
5. Inside the layer set, turn off the visibility of your overlay layer.
That should do it (again, if I understand what you’re starting with). Note that this won’t create highlights for the button, but you can do that in Encore. Select the button, then choose Object > Create Subpicture. That’ll give you a basic highlight, you can change the subpicture color sets and/or you can tweak it further in Photoshop.