Compressing gradients well is one of the hardest things a codec has to do. But I suspect you are double-compressing your material. 872x486 is not a standard for DVD authoring, as DVDs don't use square pixels, so it's possible you're encoding MPEG2 from AE, then re-encoding it in Encore.
AE is not the world's best compression tool. I recommend you do the following:
Create a new composition in AE that is a true DVD standard. Use the presets within AE. Presuming you're using Widescreen NTSC, 720 × 480 anamorphic pixels (D1 Widescreen preset).
Drag your HD comp into the new comp and resize to fit.
Render the comp to a lossless Quicktime file. The "Lossless" Output Module is a reasonable choice: it will output a lossless Quicktime Animation file.
Import the file to Encore DVD and allow Encore to do the encoding to MPEG2.
If you still see blockiness/banding, try adding a small amount of noise to the gradient in AE before rendering.
Andrew. I'm using 872x486 which is the NTSC D1 Widescreen Square Pixel preset. I don't think Encore is reencoding the video because the build process happens super fast, in less than 30 seconds.
In any case, I will try your quicktime suggestion.
Thanks for your help.
^ You may be correct, in that perhaps Adobe Media Encoder is smart enough to convert your square pixel comp to the required D1 settings for DVD MPEG2 encoding.
However, using the workflow and preset I described above is a much more reliable and efficient workflow for DVD authoring, as you're creating a file that is pixel-for-pixel created for DVD. Rendering square pixels simply adds an extra conversion step in the process.
I've seen a tutorial here and there that says increasing the noise level in the gradient can help with smoothing within After Effects. That may help reduce the blocking at least a little by the time you compress it for a DVD.
Quantization, a function of most of the algorithms used to compress, looks for pixels that are different from adjacent pixels and attempts to decide how to average them out into blocks or superblocks. If you add anything that confuses the algorithm, like noise or blur or transparency, it creates more unique blocks and that appears to keep you gradients smooth. There's a price, you give up some efficiencies in the compression processing.