6 Replies Latest reply on Aug 25, 2015 12:28 AM by deansawyer

    Mocha green screen replacement & tracking with perspective problem


      So I have this footage;



      1. Which requires the phone to be tracked and the screen replaced. I'm familiar with mocha AE and can do the easy tracking but completely lose the tracking when the phone is picked up at the start and when the top left edge of the phone goes off screen. Any idea how I can fix this tracking issue? I actually have a few more shots like this that have perspective and thumbs/fingers obscuring the phone, so none of the shots are very straight forward. I just can't seem to find any tutorials that deal with this subject in depth.


      2. I've used Keylight to key out the green screen (workflow; Key colour, - switch to status view, clip black, adjust screen gain, clip white) but I've had to clip the white so much that the footage on the guys hand now dances around, like it's noise. ie almost like the layer is on 'dancing dissolve' mode. Well I assume this is because I've clipped the white so much... Some advice on this issue would help me heaps too!


      Any help would be appreciated!!



        • 1. Re: Mocha green screen replacement & tracking with perspective problem
          Mylenium Most Valuable Participant

          Are you using exclusion masks to hide the fingers from the analysis? Have you considered just using a ridiculously large plane including off-phone areas? Otherwise you may simply need to spend some time manually re-adjusting the track. Not everything can be perfect with a magic button, especially when as per your description it was not shot accordingly. Same for your keying issue - you simply may need to get used to the thought that it requires masking, not keying. If you can provide screenshots, people may be able to provide alternative workflow suggestions, but I'm not optimistic that this can be solved easily.



          • 2. Re: Mocha green screen replacement & tracking with perspective problem
            Rick Gerard Adobe Community Professional & MVP

            First, next time you shoot something like that don't make the screens green. Turn them off or load up a dark gray to black image that you can use to capture and use the reflections. Second, there are a ton of tutorials on the Mocha website for screen replacement that is exactly like yours. In the example shot I would draw an X spline around the upper right corner of the phone taking into account a fair amount of extra area, then I would add an X spline to the lower left, then the middle, then I would align my surface and track. Once the surface is tracked I would go in and adjust the track so the surface tracks as perfectly as it can. Then I would lock the first layer (probably named track) then add a precise X spline as a second layer that precisely cuts out or roto's the screen and lock that layer to the first track. Then you make fine tuning adjustments to that track. Now you export corner pin data from the first layer (the track) and you export mask data from the second layer (roto). You make your replacement screen fit the comp, which would distort it, then pre-compose it moving all attributes, then apply corner pin to the pre-comped replacement screen. Then you use the roto layer as a mask or Mocha shape layer. The original footage is duplicated and so is the shape layer. You place now arrange your comp with the first copy of the roto layer on top, the duplicate of the original footage (oops - forgot that you shot with a green screen so this part doesn't make sense), then the second copy of the roto layer, then the corner pined replacement screen, then the original footage. Now you set the first roto layer as an alpha track matte for the duplicate footage (if you had shot with a dark screen) and set the blend mode to screen, overlay or add to add the reflections to the shot, set the second roto layer as an alpha track matte for the replacement screen to cut out the hand, make your final color adjustments and you're done. The whole key is getting a really good track and a really good roto in Mocha.

            • 3. Re: Mocha green screen replacement & tracking with perspective problem
              deansawyer Level 1

              Thanks for the replies; in regards to the tracking I've largely given up, having spent all day researching and trying to fix. I tried both of your responses but couldn't get a good track on either of them. I'm going to have to hire someone to fix these problems for me and solve the problem when I have a better understanding of mocha. As far as I understand the problem is that mocha doesn't have enough data in the planar field to work with. I thought the green screen would be enough with it's high contrast, (perhaps this is why there are track points usually on green screens!) I tried a manual track but this just leads to a jittery screen on the phone. Feeling defeated on this one, well for now at least!


              As for the keying, I spent all day researching the tracking so didn't have the time to address this fully. However the solution I found was to basically make the key a complete black and white image from the status option in keylight. This is what I read you were supposed to do and in fact is what I did do, but then because (I assume) i'd cranked the black and white values up and down so much I got that 'dancing dissolve' effect on the video that wasn't keyed. Here's how I got around it

              Dropbox - Green Screen Key.png

              Essentially creating an alpha channel from the keylights status results (which is what I thought the keying did) by using the luma track mattes. It worked in the end which is all that mattered, but i'll need to figure how this is supposed to be done properly for the future.


              Thanks again

              • 4. Re: Mocha green screen replacement & tracking with perspective problem
                Rick Gerard Adobe Community Professional & MVP

                I took your footage and brought it into After Effects.  It was very small so I doubled the comp size, then added some exposure compensation and detail preserving upscale to give me more pixels to work with. I then exported an image sequence to use in Mocha. The image sequence was imported into AE and a new comp created that matched it. I then selected Track in Mocha.


                After analyzing the footage I realized that I was going to have problems in the first few frames because of the motion blur and depth of field. There's a black bar at the top and bottom of your footage and that was also going to cause a problem. The solution is to create an X spline first to act as a hold out for the tracker.  The second step was to create a set of X splines on a new layer that covered the sides, top, and bottom of the phone. I started on a frame that showed most of the phone. The hold out layer was on the top, the tracker layer below. I tracked forward, then repositioned the CTI  to the start of the track and tracked backwards until the track started to fall apart.   Backing up to the last good tract frame I change to adjust track and manually completed the tracing. This only took about for keyframes. I then adjusted the surface and fine-tuned the track using the 16 x 16 grid as an overlay.  The track was locked and I added a new X spline to use as a Roto for the thumb. I only Isolated the part of the thumb that covered the screen.  The Roto layer was locked to the track then adjusted and edge feathering was applied.


                When that was complete I first exported the mocha shape data for the Roto layer.  Back in AE I duplicated the footage layer and pasted the mocha shape data to the duplicate which gave me a perfectly masked copy of the thumb.


                The last step was to take the replacement screen graphic and add it to the timeline then fit it to the composition and then pre-compose. Back in Mocha I copied the corner pin data from the track layer and pasted it to the screen replacement pre-comp.  By arranging the layers with the original footage on the bottom, the screen replacement next and the thumb on top the set up the composite for final adjustments. Adjusting the motion blur and focus on the screen replacement completed the project. Total time about 15 minutes.

                Your footage would actually make a good tutorial because of its problems. With your permission, if the footage is yours, I will do that when I have the time. The point is that you have to analyze your footage and then decide on the best technique to accomplish your goal. Even really horrible footage can be used. You have to analyze footage and then decide on the best technique to accomplish your goal. Just about any footage can be manually tracked in a reasonable amount of time using the right technique. All it really took to successfully track your footage was to improve the contrast and exposure for tracking and do a little work by hand.

                • 5. Re: Mocha green screen replacement & tracking with perspective problem
                  Martin Brennand Level 1

                  To fix the black bar problem without using hold out masks you can use the "Clip mask" in the Clip tab to isolate the screen to just the footage:

                  1. Go to the Clip tab
                  2. Go the Display subtab
                  3. Either adjust the numerical fields, or use your mouse and adjust the dotted box in the clip view to crop out the black bars

                  Now when you switch back to the track view you'll only see the footage area, not the letterboxing.

                  • 6. Re: Mocha green screen replacement & tracking with perspective problem
                    deansawyer Level 1

                    Sorry for the late reply, I 'm almost done with the footage; your help was very much appreciated, in the end I just stumbled through and managed to get away with it. I'll take some time out and learn how to do this properly so I know for next time. I have permission for you to use the footage as a tutorial, it'll be great to actually see how you did it vs. my hacked to death method. I'll upload the HD footage next week and send you a link. Thank you again for your generous help!