If I understand correctly, what you’re asking is: When the button is pressed, how to make it enlarge until it fits the screen.
To make the motion button itself, check out Encore’s help files, there are instructions there, or you should be able to find a tutorial at one of these sites:
https://www.wrigleyvideo.com/videotutorial/tut_encore.htm
https://doogs.sitesled.com/downloads.htm
To do the other part, where the movie thumbnail zooms up to full size, you need to create a movie of the button enlarging. This is typically referred to as a transition menu.
Take the menu file (the PSD) into Premiere or After Effects, and just animate the scale and position of the video thumbnail to zoom in as you want. Export that as an AVI, bring it into Encore, and put it in a timeline. In the menu in Encore, set the link for the button to this new timeline. Then set the end action of that timeline to the timeline that you’re actually trying to link to.
There are a couple of limitations and gotchas with this setup.
* There is no way for you to control frame sync between the motion menu and the menu transition. It’s just not possible.
* In order to control frame sync between the menu transition and the actual timeline that you’re linking to, make sure to trim the frames so that the last frame of the transition and the first frame of the timeline are contiguous.
* Depending on the player you use, there will be a slight pause, and possibly a black screen, when the DVD loads the menu transition and when it loads the timeline. Nothing you can do about that, either.