Ask in the Encore forum and more generally do your homework on the possibilities and limitations of the DVD (Video) specs. The short version is that subpictures for button highlights can only have one color and one level of transparency per state and that therefore you can't work with antialiasing or fancy stuff liek semi-transparent glows and drop shadows or whatever and if your typeface is unsuitable, it will end up being a clump of pixels. Well, anyway, ask in that forum and provide screenshots and all the otehr good stuff...
Sorry. Encore is fine but there is more control over creating in Photoshop. Encore help has a section on editing menus in Photoshop and it links out to do that. That is why I posted here. I understand that it is an Encore process I am looking at so I will move post anew there. Still, I did get the part about subpicture colors and anti-aliasing. Help is not clear about the original text layer. I did put an outer glow and a drop shadow on that. Can you tell me if that is OK? It looks great in the normal state. Also how would you know if your typeface is unsuitable? In my case I left it at Adobe Garamond from the Encore library menu which makes me think it would be fine. I had previewed a few I liked better in Photoshop but once I had it cleanly sized in my menu felt best to not change the sizes with different texts. If your willing to comment more I have included a copy of my main menu.
Those glows and drop shadows will have to be outside the button groups to be baked into the background layer. Only use the plain text inside the button groups and give each state a distinct color or use a simple underline. The text as such is okay, but I'd stay away from pure reds. They are evil for video and make video look frizzy, since they whack out the colors for the undrlying YCbCr color transform of the video signal. Also perhaps darken/ tint/ desaturate the background picture/ video a bit to improve readabilty of the text...
Thanks for the great feedback. I was trying to mimic the red seen in the uniform lettering. I may be able to get away with the drop shadow and outside glow. I made another menu a while back with even more letter effects one simple highlight layer. It does work so that when the select button is on the color on that menu, blue shifted to a gold color, a clear enough indicator that the viewer was on the correct selection. I do like the idea of having not a line but a rectangular frame around the button that would highlight for selection. My time frame and experience level won't allow I'm afraid for the time needed to learn and build a graphically acceptable menu with that feature. I'm pretty far behind on getting this done as it is. I'll try for the mix of color in selection to shift the red to the blue spectrum on select and white on activated.
Sub-picture Highlights are very limited in DVD-Video. They may be only 2-bit, and can have Transparency. Styles, FX, etc. cannot be in the Sub-picture Highlights by definition. However, one CAN create the illusion that there is say "Feather." This article goes into more detail, and explains how to create some "illusions" on the Menu, regarding the limited Sub-picture Highlights:
As Mylenium points out, some elements, that "appear" to be part of a Button, might need to be graphics elements, that are NOT in the Button's Layer Set.
If you are still having issues, I would also post a screen-cap with your Layers Palette showing, and at least one of your Button's Layer Set fully twirled open, as there might be an issue with the Layer Sets, which is the necessary structure for a DVD-Video Menu Button.
I have navigated to your article (web page I saw did not have the images) and already attempted to create my menu following the isntructons. I backed off because I have a bigger edit issue. I have a sequence with a decode mpeg error with video drop out and loss of sync that I have now asked a question for on the Premiere forum. This is not a screen shot of my layers with your transparencies but the one that corresponds my menu above.
My prior menu with effects on the text layer has about 73% opacity and 40% opacity in the fill. You can see how it works from the normal to the selected state.
When I imported my Cuba menu into Encore and viewed the selected and activated states the whole thing became a mess. I think my solution to make a simple change in the states as I did in the first menu is to open the color dialog in Encore and select the best color to make a workable change as to a good blue as I can get. I'll do some color research to see if that can even work.
I wish i could see the images of how your transparency process came out. I may run out of time to use it here because I have to solve the sync problem and get this overdue project delivered, but I will try it later.