5 Replies Latest reply on Apr 1, 2011 8:46 AM by Dave LaRonde

    Help keying jagged edges

    mttsmmnds Level 1



      I'm keying some footage shot on a 5D. Everything seems to be going ok despite the footage not being shot and lit perfectly (time, budget, the usual).


      I'm having a bit of a problem with near-horizonal straight edges, I think this may be due to the quality of the footage and the fact it was shot on a dslr with its slightly flawed rolling shutter/horizontal subsampling etc.


      I've included a still for reference.


      Can anyone suggest a quickish solution to straightening this out? Or will I need to keyframe extra masks for these edges? Also, can anyone confirm this problem is partly to do with the camera and not just my photography skills?


      Many thanks.


      Screen shot 2011-04-01 at 10.37.03.png

        • 1. Re: Help keying jagged edges
          bogiesan Level 4

          What did you do to the footage after you downloaded it from the camera?

          A "5D" is a Canon? What format does it shoot? H264? Should be transcoded to a full-frame codec before attempting to post.

          I thought rolling shutter was only a problem with horizontally moving objects; one of the many reasons I shoot HD video with a high definition video camera but that's because I've got one at my disposal.



          • 2. Re: Help keying jagged edges
            mttsmmnds Level 1

            Yep, Canon 5D MK2. I think it's more to do with only capturing every other, or every third, horizontal line of pixels... or something...


            It captures as H264, and the very first thing I did was transcode to ProRes422.


            I've decided to go ahead and start keyframing an inner mask to straighten them out. Still be happy to hear any other suggestions/ideas though.

            • 3. Re: Help keying jagged edges
              Mylenium Most Valuable Participant

              I think you very much found the cause yourself - H.264 uses undersampled chroma info and that would massively show up in red areas. The way I figure it, the area between the black frame and metal casing is causing bad antialiasing and the chroma undersampling introduces colored fringes, which the keyer interprets within its tolerance range. A different BG color would obviously have helped as might have extending the black by sticking on some black tape around the perimeter of the iPad. Anyway, masking it manually doesn't seem too much of an issue, so just pursue that course of action.



              • 4. Re: Help keying jagged edges
                mttsmmnds Level 1

                It was actually a green screen, I just droped in a red solid to highlight my problem better.


                But yes, it's definitely the coloured fringes and a manual clean-up is well under way.



                • 5. Re: Help keying jagged edges
                  Dave LaRonde Level 6

                  can anyone confirm this problem is partly to do with the camera and not just my photography skills?

                  I can confirm that the camera and your lighting skills both contributed to your dilemma. 


                  Because you didn't take the time to set up the shot properly when you got it, you now pay for it in post.

                  Because a Canon 5D Mk II  records H.264 files, you have lousy color sampling (aka color resolution); I believe it's 4-2-0, which is every bit as bad as the 4-1-1 color sampling of NTSC DV.

                  Getting a good chroma key is all about the edges of the subject.  But bad color sampling scrimps on the color information available at the edges.


                  And what do you need for a good COLOR key?  Accurate color, perhaps? 


                  To see what I mean, take 15 minutes out of your day.  Watch the following video:



                  Bear in mind that your camera's color sampling is the same as the example you'll see for DV near the end of the podcast.


                  A DSLR might make nice pictures, but it isn't cut out for effects work.  If that camera has a full-time HDMI or HDSDI out on it -- that is, the resolution doesn't change when you hit the record button -- you can drop $5,000 to get a box that will let you capture video with 4-4-4 color sampling.


                  Your best bet: rotoscoping the offending parts.  That's how you're going to pay for your shooting haste in post.