Peter,
What you are seeing is a result of the way DVD players are designed and has nothing to do with Encore. You would see the same thing no matter what authoring system is used to create the disc.
Here is what is happening. The overlay highlight is superimposed on the background by a very simple keyer built into the DVD player. By “very simple” I mean really most extraordinarily unsophisticated. It is a one-bit keyer. That is to say, it is either on or off. It has no ability to interpret a gradient in the ‘alpha’ channel. In fact, there is no alpha channel in the overlay graphic. Because the key is either on off there is no way to create pixel values typically used to antialias a graphic’s edges. Therefore, you end up with pixilated highlights.
The DVD spec does allow you to adjust the opacity of the overlay to some extent. Making the overlay more transparent can sometimes mask the aliased edging of the overlay. Unfortunately, this seldom provides the right ‘look’.
One work around is to not highlight the text (which has lots of curves) but instead provide some kind of graphic primitive to serve as the highlight indicator. Underlining or bullet points such as arrows or triangles are possible alternatives. In general, avoid curved edges.
To make menus where glows, bevels or embossed looks indicate the selection requires multiple pages of menus in conjunction with auto-activated buttons. This can become a lot of work if you have many buttons because a separate menu graphic is required for every selectable button. When using this technique the overlay capability is not used at all.
Hope this is useful.
Dave