15 Replies Latest reply on Oct 31, 2009 1:18 PM by Mylenium

    AE in 32 bits mode, masks and banding

    ideami

      hi, im doing an AE CS4 project in 32 bits, i create a mask and with all levels of feathering and expansion i can see banding so clearly, i thought 32 bits would solve the banding issues, i see this with all examples, say that i put a black solid over a clip and i mask it with an oval to create a vignete, i use feathering and expansion where necessary, but still in 32 bits mode i can see clearly banding, of course with the add grain filter i can get rid of it, but i add also noise; am i doing something wrong, is it possible that in 32 bits mode banding still happens? there is non way to avoid it?

      thank u so much for any help

      jav

        • 1. Re: AE in 32 bits mode, masks and banding
          Rick Gerard Adobe Community Professional & MVP

          Are you seeing banding in the comp window as well as in the renders? Banding in the comp window is common if your display adaptor isn't set to the highest color settings.

          • 2. Re: AE in 32 bits mode, masks and banding
            ideami Level 1

            thank you very much for your answer, yes, i keep seeing it,

            i just did a render at 32 bits, and checked the quicktime rendered outside, the banding is so very clear , i wonder what can it be

            thank u so much for your help

            • 3. Re: AE in 32 bits mode, masks and banding
              ideami Level 1

              i just checked, my display adapter also is at 32 bits, so that one is ok,

              mmm,

              i was enthusiastic combining AE with color finesse to create the masks in AE, but cant understand why so much banding happens, in 32 bits

              • 4. Re: AE in 32 bits mode, masks and banding
                Mylenium Most Valuable Participant

                Well, have you verified the output on a different system? I still tend to think that your monitor simply isn't particularly suitable for such HDR imagery and merely fails to correctly dither down to 6bit (which is most LCD's effective color depth). That aside, do not forget that masks are 8bit, regardless of the project bit depth. If you exceed certain values, there will always be banding in the transparency, in turn of course affecting the RGB channels as well - simple math dictates this. In addition, simply look for other things like OpenGL usage or non-float effects breaking the rendering chain.

                 

                Mylenium

                • 5. Re: AE in 32 bits mode, masks and banding
                  ideami Level 1

                  dear Mylenium, thank you very much for your very kind and detailed answer,

                  my monitor is a viewsonic 24 inches lcd, its really interesting your point about the masks being 8 bits, i hadnt thought of that... thats so true,

                  but then, lets say i want to do a simple vignete, how do i do it to keep it all smooth?

                  and u know basically im doing color finesse corrections and i want to use AE on top to mask the effect of finesse in different areas, so isnt there a way to do that smoothly?

                  thank u again so much

                  jav

                  • 6. Re: AE in 32 bits mode, masks and banding
                    Mylenium Most Valuable Participant

                    Yes, of course this should be perfectly possible, I'm just not clear as to where it goes wrong for you. Could it be, that you are using some compressed footage type and may in fact be seeing incorrect float expansion? Have you tried applying the HDR Compander effect (in the Utilities category)? What blending mode do you use on the vignette? Is your project color managed? And what version of AE are we exactly talking about? also, do you by any chance clip the ranges in Folor Finesse? This could drastically worsen quality in 32bpc as it wil lbe more pronounced than in low bit depths...

                     

                    Mylenium

                    • 7. Re: AE in 32 bits mode, masks and banding
                      ideami Level 1

                      thank you very much Mylenium, i explain u what im using

                       

                      and also attach here 3 images where you can see the banding itself, my project settings and my R3D Raw settings

                       

                      AE version is CS4

                      Footage is RED One footage R3D Raw build 20,

                      the project is not color managed,

                      the color space of the red footage is RedSpace

                      in color finesse im not clipping the footage, i check the waveform monitors etc

                      blending mode of vignete is Add

                      basically i have a layer with the RED R3D footage, on that layer i create a mask , outside of the mask black appears, i put feathering etc, the blending mode of the mask is Add, which is the default

                       

                      i havent tried that compander effect will check that, i just tried it, interesting but cant quite manage to make it work well i think mmm

                      thank u very much again

                      jav

                       

                       

                       

                      Yes, of course this should be perfectly possible, I'm just not clear as to where it goes wrong for you. Could it be, that you are using some compressed footage type and may in fact be seeing incorrect float expansion? Have you tried applying the HDR Compander effect (in the Utilities category)? What blending mode do you use on the vignette? Is your project color managed? And what version of AE are we exactly talking about? also, do you by any chance clip the ranges in Folor Finesse? This could drastically worsen quality in 32bpc as it wil lbe more pronounced than in low bit depths...

                       

                      Mylenium

                      • 8. Re: AE in 32 bits mode, masks and banding
                        ideami Level 1

                        here i attach another picture where i have just applied a simple brightening curves command in photoshop to the screen capture just to see the banding easier, the banding is also there clearly without the curves command, i just applied it to show better the texture of it

                        jav

                        • 9. Re: AE in 32 bits mode, masks and banding
                          ideami Level 1

                          wow there is something really wrong, i just started a brand new after effects CS4 project (and by the way, yes, my software is all original, legal and bought and payed for just to make it clear), so new project, and

                          i create a solid green layer

                          and on it a make an oval mask, i make feathering and see the images attached what i get,

                          wow.jpg shows what i get in original, and wow2.jpg is the same with brightening curves applied to see better,

                          how can this happen, the project again is a 32 bits project...

                          thank u for any help

                          jav

                          • 10. Re: AE in 32 bits mode, masks and banding
                            Rick Gerard Adobe Community Professional & MVP

                            I see two potential problems. First, your resolution is set to half so you're seeing blocks of 4 pixels instead of the actual pixels. Second, try turning off Open GL rendering.

                             

                            If that doesn't fix the problem then please post your project. Even with 8 bit project settings, when going from black (r, g, and b to 0) to green (r and b to 0, g to 255 or 120º and brightness and saturation to 100%) you have all 256 steps available so a feather of 256 pixels on your mask would give you a perfectly smooth gradient.

                             

                            If that doesn't fix your problem please post your project. I can't get a simple solid and mask to foul up.

                            1 person found this helpful
                            • 11. Re: AE in 32 bits mode, masks and banding
                              ideami Level 1

                              dear Rick Gerard, thank you very much for your very helpful answer, i have tried those two things, project to full resolution and

                              opengl off but still same problem so here i post you the AE project itself, thank u again

                              jav

                              • 12. Re: AE in 32 bits mode, masks and banding
                                ideami Level 1

                                dear Rick Gerard, thank you very much for your very helpful answer, i have tried those two things, project to full resolution and

                                opengl off but still same problem, i tried to post the project here but aep extension is not allowed so i uploaded them online and here they are, i did one in after effects and one in photoshop, same problem,

                                 

                                http://www.ideami.com/tech/bandingae.rar

                                http://www.ideami.com/tech/bandingpsd.rar   

                                 

                                (in after effects and in photoshop)

                                 

                                thank u again

                                jav

                                • 13. Re: AE in 32 bits mode, masks and banding
                                  ideami Level 1

                                  really this seems now to be a generic issue of after effects, photoshop etc, lets write the facts

                                   

                                  1) im testing with two computers, a good laptop and a good desktop, both new , powerful, both with NVIDIA graphic cards which i guess are supposed to be good graphic cards

                                   

                                  2) im using the latest software, photoshop CS4 and AE CS4

                                   

                                  3) im trying all this in 32 bits in both and in the other modes later also

                                   

                                  4) i am switching off opengl as for the suggestions i receive, and also im seeing all at full resolution

                                   

                                  so given all that written above, now we go to photoshop, create a new file say 1920x1080, take a greyscale gradient with the gradient tool and put it on a new layer,

                                   

                                  and there it is, the banding! ))

                                   

                                  so either there is something really wrong in my setup, or this is a generic problem

                                   

                                  i thought it could be my video hardware, but how could this be, both my control panel settings of both computers are set to highest color depth 32 bits, and both have NVIDIA graphic cards, and both are a year old or the desktop a month old, i mean it cannot get much more recent, certainly its not the best hardware around thats for sure, but really i think it should be enough to display an smooth gradient i would hope

                                   

                                  i have known about the technique to use Add Noise for a long time to get rid of banding but i didnt remember that banding would happen so easily in a simple gradient or in a simple mask even at 32 bits, this is what puzzles me, i thought the whole point of 32 bits is that these things like this type of banding wouldnt happen

                                   

                                  thank u for any help

                                   

                                  by the way ,searching around online i have seen so many threads of people complaining about banding,

                                  is Adobe addressing somewhere what seems to be a generic concern about Banding, maybe with a nice article explaining the reasons, causes, and solutions for banding that u may still get in 32 bits etc? that would be very nice

                                   

                                  thank u again

                                   

                                  jav

                                  • 14. Re: AE in 32 bits mode, masks and banding
                                    Bob Currier Level 3

                                    One cause of this can be monitor calibration. If you have calibrated your monitor and it has had to push the curves very much at all to bring things into calibration, you can lose a lot of available pixel codes. It's usually not a big problem, but if you had to make any extreme adjustments then you will definitely see banding later on, independent of what any application does.

                                     

                                    One way to check for this is to look at the pixel values in the buffer before they are displayed. Within AE (or PS) take one of your examples of bad banding and examine the pixel values in the area of one of the band transitions. If what you find are that the pixel values smoothly increase or decrease, but what you see is a big step, then you need to look at issues outside of the application, such as calibration, bit depth of the LCD panel, etc. It's easiest to do this if you look at only one of the color channels. You should see it regularly increase or decrease as the mask feathers out, regardless of how it looks on the screen.

                                    • 15. Re: AE in 32 bits mode, masks and banding
                                      Mylenium Most Valuable Participant

                                      I'm not seeing banding. That is, beyond what my monitor would produce based on its limited gamut/ color range. The only thing of note in your AE project is, that you set the feathering way beyond what the radius of your mask can accomodate, which makes it run twice over some areas, intensifying the problem. The rest looks pretty normal, given, that such pure colors are particulalrly prone to showing on-screen banding due to how they only activate one light dot in your RGB triplet cells. It#s realyl more an issue of how this reduces the perceived color range rather than anything being wrong with the project itself.

                                       

                                      Mylenium