There is no way to do what you want by using subpictures to highlight a button. As you have noticed overlays are always hard-edged.
The effect you want requires multiple versions of your menu – literally one menu page for every glow/blur combination you desire. You then use buttons that Auto Activate and route to the appropriate menu page. (see Encore’s help files for info on auto activate)
This can get pretty tedious if you have a lot of buttons. It’s even more complex if you want to indicate that a button has been activated in addition to being in the selected state.
The hard-edge nature of subpictures is due to the nature of the DVD spec and the hardware that accomplishes the overlay in DVD players. It is not an Encore imposed limitation.
Hope this helps.
Dave