11 Replies Latest reply on Jul 19, 2016 2:15 AM by lornaimw

    Rotoscoping edges and clean up

    lornaimw Level 1

      I'm rotoing 6x10second clips and its been a bit of a nightmare.. none of them look perfect and days have been spent on it.

       

      However there is one I'm having a real problem with that looks like the roto is good, but as soon as I click out to the composition, everything falls apart and its all dotty.

       

      What is this and how can I clean it up, its fine in the source viewer viewing the roto tool etc, as you can see in the screen grabs, but as soon as I change tabs and go to the composition it's actually held in, it changes. I can't understand why. This comes out on the render so it really must be fixed.. This happens even if I create a new composition and drop it in...

       

      Any advice?

       

      Or even any advice on generally cleaning up rotos? Im really struggling.

       

      I'm using AE cc2015.0.1Capture.JPGCapture1.JPGCapture2.JPG

        • 1. Re: Rotoscoping edges and clean up
          Rick Gerard Adobe Community Professional & MVP

          If the blue is in the original shot I'd key this shot and not roto it. You'll get a better result in a lot less time. Biggest problem is that the shot as I look at it is not precisely in focus or the frame you have chosen has motion blur.

          • 2. Re: Rotoscoping edges and clean up
            lornaimw Level 1

            Rotoscoping is the only way I can see.. or masked rotos.

            The one in blue is just the alpha overlay (shown as blue for me in my settings as it had the most contrast)

             

            The wall is actually yellow ( a complete nightmare to key out!)

             

            I've had 6 of these to do.. on white and purple.. it's been interesting.

             

            Any other tips on cleaning up rotos? You know how it leaves behind too much chatter or black edge?

            • 3. Re: Rotoscoping edges and clean up
              Rick Gerard Adobe Community Professional & MVP

              Post a frame of the original footage. If the wall is relatively uniform in color I would try to create a procedural matte using colorama or channels and then combine that with some simple roto or basic masks. Roto or rotobrush is a last resort tool for this kind of shot when nothing else will work.

              • 4. Re: Rotoscoping edges and clean up
                StormyCPb Level 3

                Your edges should not look like that if your matte in the top looks like it does. Are you sure you've not set the layer to 'Dissolve' by accident?

                • 5. Re: Rotoscoping edges and clean up
                  lornaimw Level 1

                  Original footage looks like this (this is the actual orientation I'm working at)

                   

                   

                  I've always struggled using/ creating something like a procedural matte. But in this instance I've used rotobrush and masking and matte chokers etc to help finesse. I ended up rendering the file on another machine where it didn't have this dotted edge but it is really random.. and StormyCPb If the layer was set to dissolve it would show the entire layer as such.. not just the edges surely?

                   

                  Edited to include photo! (Woops)2015-08-04_MockUpScreengrabs HD rotation_[#####BOY]_00219.jpg

                  • 6. Re: Rotoscoping edges and clean up
                    Rick Gerard Adobe Community Professional & MVP

                    I would roto the ball by hand and use that as a hold out mask, then use color adjustments, maybe even colorama to smooth out and change the yellow background, then use Keylight to create a matte by showing the completed matte. That layer would be pre-composed and used as a luma track matte for the original footage. I would also add key light and sample the yellow background color to remove some of the color fringing while letting the luma matte take care of the separation.

                     

                    If the screenshot is accurately representing the footage then it's not very sharp so you're going to have to do some light wrap or other tricks to hide the soft edges, especially in the hair and the motion blur areas.

                    • 7. Re: Rotoscoping edges and clean up
                      StormyCPb Level 3

                      Surely? No... If it's set to dissolve at 100% Opacity then your edges would look rough just like yours does, which is why I suggested it.

                      • 8. Re: Rotoscoping edges and clean up
                        StormyCPb Level 3

                        Your modes are hidden so I can't tell...

                        • 9. Re: Rotoscoping edges and clean up
                          StormyCPb Level 3

                          I'd probably try using the RGB channels to create luma mattes first. There's some contrast with the arms and face on the blue channel, the t-shirt could be lifted from the the green, and the hair from the red, then roto the ball. If the image is a good representation of the footage then I'd ditch keying and the rotobrush, it's just too noisy and awkward.

                          • 10. Re: Rotoscoping edges and clean up
                            lornaimw Level 1

                            The screen shot is a jpg, so compressed. It was shot in 4K from a sony fs7 in rec709, this is an HD snap of a 4k piece of footage.

                             

                            The results I've got from the rotobrush were OK, I have no idea what the problem with the edges was but thanks for your advice on the other mattes. Very good to know how you could do this in a less time consuming way with hopefully a better finish.

                            • 11. Re: Rotoscoping edges and clean up
                              lornaimw Level 1

                              Interesting to hear yours and Ricks idea's on how you would complete the task of taking the yellow wall out, food for thought if I have this sort of obstacle again. I put dissolve on the clip and you're right, it does look like that. I generally have no need for that mode and tend to ignore the ones I don't use. I thought it would make the whole layer dotty. Good to know!

                               

                              Thanks all

                              • 12. Re: Rotoscoping edges and clean up
                                Rick Gerard Adobe Community Professional & MVP

                                There's so much compression in the screen shot that I'm getting lots of color artifacts but here's how I would attack the problem. First I would roto the balll using mocha. Just the ball with soft edges. You might even be able to do that with AE's mask tracking.

                                 

                                On a second layer I would enable color correction and use levels and key light to create a plate with a decent key. It would look something like this:

                                (Key light turned off so you can see the color adjustments)

                                Screen Shot 2015-08-26 at 7.06.24 AM.png

                                The idea of the color correction is to get as much separation as possible between the skin tones and then wall that needs to be keyed.

                                Then Keylight is applied, the ball layer is placed on top, and the original footage with some quick masks to hold out the stuff that the keyed middle layer can't get rid of. As a last step Keylight is applied just to take care of some of the spill. Other than the jpeg artifacts I can get a pretty decent key using this technique. A white solid was used on the bottom to emphasize the edge color problems.

                                Screen Shot 2015-08-26 at 7.46.27 AM.png

                                1 person found this helpful
                                • 13. Re: Rotoscoping edges and clean up
                                  lornaimw Level 1

                                  Thanks for this response Rick. I didn't get the time to try this on the project but will test it myself when I get the chance. Would be very interesting to see how well it could work. Thanks for your help!