6 Replies Latest reply on Sep 12, 2016 11:36 AM by thomasf38949141

    Remove semitransparent foreground elements - Thoughts on how to approach this

    thomasf38949141

      Hello,

       

      this should not be classified as another "How do I remove watermarks from my footage" discussion. However I will initially use this exact "problem" to illustrate my point and get to serious compositing situations later on.

       

      We stay with the watermarks for now.

      Let's assume that we want to remove a watermark or another semi-transparent object from a video. Content aware fill, blurring or similar approaches won't do the job convincingly. So the next thing we can think of is to recreate the watermark, put it on top of our footage and somehow use arithmetic blending modes like "subtract" or a color-difference key in order to make it disappear. But none of those ways offer the required math unfortunately. I created a test project where I added a watermark (a simple shape with 30% opacity) over a video. I precomped both as "footage", and then used another instance of that watermark on top of that to negate the first nested watermark again. I basically tried to restore the original image with blending modes and the color-difference key but failed. You're still able to identify the pixels that were affected by the process of adding and subtracting the watermark. They just stood out with difference in brightness, contrast or color. Of course you can tweak results with color correction. But well, this is more trial and error than a sophisticated and elegant way of doing it.

       

      footage_comp.jpgwatermark_removal_comp.jpg

       

      It feels like there should be a feature or workflow for this. Something like a negative/backwards compose blending mode, which can - in my example - be used on the top layer (the second copy of the watermark) to blend out its counterpart from the precomposed layer below. So the top watermark layer on which the blending mode is active, just tells the program what unwanted color had been added to the footage below. And it reverses this blending operation to restore the original image.

       

      You could go even further and do this for more blending modes. Maybe you notice that the watermark was put over the video using the blend mode "add". So there can be an option that lets the user chose, which blend mode was used and should be negated.

       

      I don't know if I am oversimplifying this but I guess all parameters for this in order to work are there. We have the watermark affected footage on the one hand. On the other hand the top layer provides information on the watermarks shape, color and opacity. Can't we use this to undo the blending operation with some math?

       

      I guess the quality of the result would depend on how well you could recreate the watermark/object. And of course color quantization would increasingly appear, when the opacity of the watermark was high.

       

       

      Where could this be useful?

       

      This could be a great alternative to content aware fill when dealing with unwanted semi-transparent foreground objects like lens flares, lens dust, even windows that you shot through etc. In some cases you could just roughly repaint those objects on a seperate layer with this blend mode active and watch them disappear.

       

      Sometimes, you might even be able to use the foreground objects themselves without having to repaint them. This example is the best I could come up with:

       

      Let's assume your lens was dirty when you finally got your eagerly planned 2minutes long take right. So instead of doing it the 56th time you try to fight the dirt in post-production. Of course you can't clone stamp every frame so you use the same camera to make a photo in front of a greenscreen to get the same dirt on a transparent backgound. When something like this blending mode existed, you could now use this layer on top of your footage to get rid of the dust.

       

       

      Let me know if something like this already exists or my thinking process has logical flaws. There obviously has to be something difficult about this. I am sure they would have implemented it otherwise.

        • 1. Re: Remove semitransparent foreground elements - Thoughts on how to approach this
          Rick Gerard Adobe Community Professional & MVP

          You're thinking is flawed. If you wanted to use a blend mode or something similar then the copy you put over the originals would have to know the difference  between the values of the two original source files.  The only nearly automatic way I know of to remove object from video is to use a time displacement algorithm for moving pixels or clone in disguise the fix.

          • 2. Re: Remove semitransparent foreground elements - Thoughts on how to approach this
            thomasf38949141 Level 1

            But Rick, in my example I know the exact color and opacity values that the watermark added over the fractal noise. And I should be able to reverse the blend calculation, as long as I can bring the exact object shape, color and opacity to the table. This process should only be limited by quantization.

             

            And the idea is that if I am unaware of those exact watermark variables I can simply "trial and error"-change them on the top layer, to get as near as possible to the original watermark, which would lead to a more or less good removal.

            • 3. Re: Remove semitransparent foreground elements - Thoughts on how to approach this
              Mylenium Most Valuable Participant

              So the next thing we can think of is to recreate the watermark, put it on top of our footage and somehow use arithmetic blending modes like "subtract" or a color-difference key in order to make it disappear. But none of those ways offer the required math unfortunately.

               

              How should there be? You are completely not understanding the meaning of blending modes. After the blending operation, the original color is replaced with the new color and since for any given pixel the same color can be created with different modes, there is simply no 100% way of restoring the original color. It's mathematically ambiguous, especially since many blending modes use non-linear formulas and normalize the values internally, as a result of which the output is always biased and deviates from the input values even more and is many times sensitive to the already weighted color components. On top of that you have to deal with Gamma and other things. I also don't see how one would even store the original color in some hidden form in the result color. The value ranges of conventional 8bit and 16bit integer would at best compress the remaining space, definitely at least causing quantization deviation and even in float formats you could not simply assume that the output value is the linearly added result of color A as the pedestal/ base offsett and on top color B is stored. That's simply not how it works, especially once you move beyond simple single-channel (greyscale) operations. That doesn't mean it may not possible to reconstruct parts of the info in some way, but most definitely simply using the opposite blending operation isn't it. If at all, this would require complex algorithms that make educated guesses and analyse the surroundings to figure out which source colors may have been involved, but other than that your thinking is completely flawed. You want something that is impossible.

               

              Mylenium

              • 4. Re: Remove semitransparent foreground elements - Thoughts on how to approach this
                thomasf38949141 Level 1

                Thank you for your answer! But please let me explain again, just focusing on the normal blend mode. I think I still don't get it 100%.

                Mylenium schrieb:

                 

                How should there be? You are completely not understanding the meaning of blending modes. After the blending operation, the original color is replaced with the new color and since for any given pixel the same color can be created with different modes,

                Yeah, I'm aware of that. That's why the user would have to input the information on how the color was created, in setting the parameters of the top layer. And colors that were created with blend modes that consist of if/else statements can't be reversed. It's impossible there.

                 

                there is simply no 100% way of restoring the original color. It's mathematically ambiguous, especially since many blending modes use non-linear formulas and normalize the values internally, as a result of which the output is always biased and deviates from the input values even more and is many sensitive to the already weighted color components. On top of that you have to deal with Gamm and other things.

                 

                What I still don't get is why you can't restore the color of the normal blending mode:

                 

                Normal Blend Mode:

                Darby Edelen on CreativeCow schrieb:

                 

                If you assume 'A' is the top layer, 'B' is the lower layer and their alphas are 'a' and 'b' respectively then the resulting color 'D' is:

                 

                D=A*a + B*b*(1-a)

                Now when I Input the information I have:

                 

                D=Footage with watermark

                A=watermark  (set by designing the top layer - I have to guess this value when I don't have the exact information as in my example)

                a=watermark alpha (set by changing the top layers opacity - I have to guess this value when I don't have the exact information as in my example)

                B=original footage pixel values

                b= original footage alpha (always 1.0 - this only works when the original footage has 100% opacity)

                 

                Still not sure why I can't solve the formula to something like:

                B=...

                 

                Mylenium schrieb:

                I also don't see how one would even store the original color in some hidden form in the result color.

                 

                Nah. That's not the plan.

                • 5. Re: Remove semitransparent foreground elements - Thoughts on how to approach this
                  Mylenium Most Valuable Participant

                  What I still don't get is why you can't restore the color of the normal blending mode:

                   

                  Normal Blend Mode:

                  Darby Edelen on CreativeCow schrieb:

                   

                  If you assume 'A' is the top layer, 'B' is the lower layer and their alphas are 'a' and 'b' respectively then the resulting color 'D' is:

                   

                  D=A*a + B*b*(1-a)

                   

                  Why should you? That formula is of no use and on some level wrong. The operations are applied per channel/ per pixel, not as a compound. Multiplying e.g. the red color component via the Alpha value on a semi-transparent pixel against a blue background would have another output value as the same color multiplied against green. The other color components will behave similarly. On top of that you would have to figure in sub-pixel sampling and at least the layer's opacity value, which the formula doesn't account for. In the end, you end up with too many variables. That's nothing specific to AE, that's a simple math thing. You simply never will know if a dark red color you see was created by fading a semi-tranparent black pixel on top of an opacke red one or whether its a combination of partially transparent pixels that incidentally happen to add up to 100% opacity. Such is the nature of the thing and even if you know some of the variables, it remains an ambiguous equation that has multiple solutions.

                   

                  Mylenium

                  • 6. Re: Remove semitransparent foreground elements - Thoughts on how to approach this
                    thomasf38949141 Level 1

                    I may sound ignorant, but I think I we are still not fully understanding each other. Of course this works per pixel and per channel and I think it could work as a blending mode. Or am I still blinded by something and won't notice it?^^ Here is a detailed calculation example:

                     

                    Generally we don't have the information on how the pixel colors of the watermarked footage had been calculated. But for now let's assume it was something like this for one random example pixel:

                     

                    original footage pixel (without watermark): rgba(45,82,112,1)

                    watermark: rgba( 220,150,110,0.3)

                     

                    So the final watermarked footage pixel would be

                    r=220*0.3+45*1*(1-0.3) = 66+31,5 = 98(rounded)

                    g=150*0.3+*82*1*(1-0.3) = 45+57,4 = 102(rounded)

                    b=110*0.3+112*1*(1-0.3) = 33+78,4 = 111(rounded)

                     

                    -> watermarked footage: rgba(98,102,111,1)

                     

                     

                    Now think about it the other way around. We start with the watermarked footage and have no reliable information on how the color of the pixel was, before the watermark had been added. So the original footage pixel and the overlaying watermark pixel are unknown at first. I agree with that of course.

                     

                    Here comes trial and error with live-feedback (as with any other transfer mode) into play. You basically have to guess the right color and opacity values of the watermark until it disappears. Here's how:

                     

                    When you solve the formula of the normal blend mode from:

                    D=A*a+B*b*(1-a)

                    to

                    B=(D-A*a)/b*(1-a)

                     

                    you get the pixel value of the original footage, without watermark (B)

                     

                    which means in this case:

                     

                    watermarked footage: rgba(98,102,111,1)

                    watermark (guessed with live feedback): rgba(220,150,110,0.3)

                     

                    r=(98-220*0.3)/1*(1-0.3) = (98-66)/0.7 = 45(rounded)

                    g=(102-150*0.3)/1*(1-0.3) = (102-45)/0.7 = 81(rounded)

                    b=(111-110*0.3)/1*(1-0.3) = (111-33)/0.7 = 111(rounded)

                     

                    Recalculated original footage pixel: rgba(45,81,11,1)

                     

                    Pretty close!

                    ...now you will notice that the calculation of the value for the green channel is slightly off. (81 instead of 82). This is the quantization, I was talking about, which will increasingly happen, when the opacity of the watermark gets higher.

                     

                    But anyway. We roughly calculated the color value of the original footage pixel.

                     

                     

                    Now you may think that it is pretty unlikely to guess the values of the watermark right. But it is not. When you have live feedback and you can scrub through the values and see whether the  watemark gets more transparent or more opaque, you can get a good feeling for the values. So it may not be very problematic. And even if you don't guess 100% correctly you may at least be able to visually make it vanish.

                     

                    ...and I don't see why this should not work for some other blend modes too. Just solve the formula to "B=...". Not on all possible though.