If you have a good encoder VBR unconstrained 9-pass would be about the best you can get. AME does not give that option.
With the options that AME gives you, it depends on the length of your sequence, the nature of your movie (slow or fast action, pans and zooms), the time you want to invest and the desired delivery format. As a general rule of thumb, the results from this calculator gives you nice results: DVD-HQ Bitrate & GOP calculator
Keep in mind that encoding is an art. Whole teams of encoders are employed in Hollywood just to make the best of their movies.