Banding will happen in 8bit renders if there is not enough room in the color values to make smooth transitions. For example if you have a blue gradient that covers 250 pixels and the blue value changes from 200 to 250 you are going to have 10 bands of color in the gradient. If your preview is full resolution and the Magnification factor is 100% you will see that banding if you look closely. When you render to MPEG compression the banding problem increases especially in smooth areas of color transitions because great big blocks of pixels are averaged together and you only get a few frames of color information that represent the actual pixels in the project. You can try and improve the color artifacts in the Adobe Meida Encoder by checking the highs quality options in color and you can also reduce the color artifacts by enabling multi pass rendering and increasing the data rate a bit.
The best practice is to add a little noise to the shot so the MPEG compression has more data that is changing to work with. This will tend to hide the bands by breaking up generally straight lines between color values.
If you are doing heavy color grading you should be setting your project to 16 or better yet 32 bit color. This will help eliminate banding in 8 bit renders. Unless you have a 10 bit or better monitor and a GPU capable of driving it it's really hard to judge banding by looking at a comp preview. The info panel should show the right color information but it's very hard to actually see it in the monitor, especially if you are judging a frame at anything less than full resolution and 100%. The perfectly blended frames are probably artifacts or your eye is being tricked. To see exactly what's going on in the frame you need to purge the image cache. Depending on your graphics card your preferences may be fouling up the previews by trying to build higher quality previews than actually exist. You could also be getting some funky data from the original video if the source is a highly compressed MPEG stream. The CPU may also be adding to the preview looking OK when the values are not really there because it is averaging the small amount of color and luminance values in the "master" frames and the "gee I'm not really a frame but here's a little bit of data about where I think the pixels are going" frames. Remember the I'm not really a frame can out number the real frames by 2 or 4 to one. The actual color information even in the real frames may be compressed averaged into big blocks as well. All of this contributes to problems in subtle color changes. That's why noise helps.
Hope this helps.
Some really helpful stuff, thanks a lot for the thorough answer.
I still don't really understand how after effects was able to create these perfect gradient versions of my processed low light footage, as a glitch. I wasn't aware of a lot of the potential reasons you mentioned like the non-frames, though this glitch was occurring on any frame I landed on. I would guess this was because I was using blur effects and maybe they were being processed at a higher depth maybe, even though my project was definitely set to 8 bits.
Also thanks a lot for letting me know about the image cache purge, I forgot about that feature..
You mentioned to work with 16+ bit depth in the project, however when it comes to encoding the output, will the banding not occur anyway when its encoded to an 8bit video? This is something I keep meaning to read up on.. is it possible to encode higher than 8 bits, if so is it compatible with most media players etc, if not how is colour info processed/filtered.
Thank you a lot for the heads up about ticking the maximum render quality, I assume ticking "render at maximum depth" will help too?
I'm using h.264 and managing to get smooth playback at around 4mb min 8mb max 1vbr.
I know about the noise and haze alternatives to get rid of banding, and in the end I just ended up brightening the footage because there was so little colour information to work with.