5 Replies Latest reply on Sep 8, 2015 5:54 AM by Iggorr25

    How to track for 3D model


      Hi Ok so I want to work on a new project I have a plastic gun I have tried tracking the gun by placing 4 white paper peaces on it, and using that data to replace the gun with a 3D model of a gun in Element 3D. I did not work.... How can I do this? I want to pretend that I am shooting with this plastic gun and then replace it with a 3D model of a gun in Element 3D so I can animate the gun shooting like a real one and create a cool scene that I have in mind, I know how to animate a gunshot with masking and stuff like that but with what I have in mind I need the 3D model of the gun in my hand........ any tips, link maybe for a tutorial on something like this?

        • 1. Re: How to track for 3D model
          Szalam Adobe Community Professional & MVP

          How your camera moves in the scene and how your hand moves are pretty crucial for giving advice. Could we see the shot?

          • 2. Re: How to track for 3D model
            Iggorr25 Level 1

            I have not recorded the clip yet, but here is a quick one I just made to try and better explain what I want to do. You can see the plastic gun I have and the 4 white squares I want to use as tracking marker. Now the idea is that I will do this video with a green screen in the back so I can end up with a black background, or no background. I need to replace the gun with a 3D model of a gun inside of the plugin Element 3D from videocopilot. After that I will animate the slide of the gun to follow my hand as if it were being pulled back, at this point I will freeze the video and want to zoom on the gun and show the bullet from the top inside of the gun, now unfreeze the video to show the bullet being loaded. Then keyframe the bullet flying out of the gun and make the camera follow the bullet which is also a 3D bullet to show how it hits a target revealing something. So basically this is what I want to do but with much more detail, And I have animated the gun already just to test it and it turned out great. I need to replace the plastic gun with the 3D model one, and mask out the part of my hand that will hold the gun. The camera will be probably on a tripod.      test - YouTube

            P.S. All the camera movement I was talking about is going to be keyframed with the 3D camera within AE.....

            • 3. Re: How to track for 3D model
              Rick Gerard Adobe Community Professional & MVP

              The tracker that comes with AE and the version of Mocha that is included will not give you the kind of tracking data that you need to replace the gun in your hand with a 3D mode. Let's take your example video and break down what you would need to do to just replace that with a 2D image of a gun. I'll give you my take on your idea and your technique.

              Screen Shot 2015-09-04 at 3.07.22 PM.png

              First Let's talk about the tracking markers. For good tracking with any of AE's trackers you need detail. Your white against black only has edge detail and it's not going to be very easy to track. If I were sticking something on the gun to track I would stick something that looked like a QR code with lot's of detail. If you gave me this shot to track I would probably grab one end of the barrel and the back of the gun just above the left thumb.


              Next let's talk about the shadows that the hand makes. If you are going to replace the image of the gun you're going to have to figure out how generate some shadows to sell the shot. You're also going to have to roto out the part of the hand that covers the gun. In this particular shot it's probably going to be easier to do the roto by hand. You can then separate the layers so you'll have something to put on top of the replacement gun.


              Now let's talk about replacing the gun. First, the replacement footage can be from anything. You can use a 3D model. If the gun fires you don't want to have the footage of the model move at all. The footage will also have to be taken or created from the same camera position in relation to the gun. Focal length crops the image, camera position creates perspective. So once you film the gun at the same angle you can then track the gun or the hand, then apply that tracking data to the gun footage, then put the shadow layer you created next and finally the copy of the hand that covers the gun.


              If it were me doing this shot I would stabilize the shot using AE's native tracker so that I locked the top part of the gun into position, then I would duplicate the footage and roto the hand and roto a shadow layer, then I would position the replacement gun footage, build my composite, then add a null and use my destabilize animation preset to add and expression to the null. The last step would be to parent all of the layers to the null. This would remove the stabilization from the stabilized layers and add motion to the gun footage. If everything worked just perfectly and your pantomime of pulling back the receiver was dead on accurate then the shot might be convincing.


              If you have a perspective change on the gun. Let's say that you start the shot with the gun pointing at the camera and then you swing the gun to the left so that it matches the position of the gun in your sample shot then you have to recreate the 3D geometry of the gun and hand, build 3D models for both and then composite your replacement gun into the scene. This is way beyond what comes with AE. You'll need something like Syntheses or maybe Photoscan to generate your 3D info.


              Here's what my replacement project would look like: I spent about 15 minutes getting to this point.

              Screen Shot 2015-09-04 at 4.17.21 PM.png

              Just getting this far I ran into several problems. First the gun's are not exactly the same shape so there are problems overlaying the new gun especially in the trigger area. Second, in the shot the gun is tilted from the vertical a couple of times so the perspective changes giving away the replacement gun. Third, look at the tracking keyframes in the bottom layer. See that they end before the shot does. This is because the gun moved so fast that the motion blur meant that this part would have had to be tracked by hand. That was more work than I wanted to do on this example. If there was a clean plate, a shot without the hands in the shot, and the shot was filmed either at a high frame rate or a little more carefully, and the gun did not change angles, with a little color correction I would have been able to pull this off as believable. I only rooted the first 4  seconds


              So here's my final analysis of your proposed project. To make this work inside AE you'll need to be very careful to keep the gun at exactly the same angle to the camera so the perspective does not change, you'll need a clean plate, you'll need to have footage of a gun that accurately matches the profile of the original gun, and you'll have to be very careful with the hand movements.


              To do the shot using a 3D app and generating a 3D model of the hand and gun so you can drop in a 3D gun is going to require you to also very carefully plan and execute the shot considering all of the limitations in the 3D software you will be using.


              My suggestion, buy or rent an AirSoft gun and add a muzzle flash. Getting the shot you supplied to work perfectly would require at least a half day with an experienced user.

              • 4. Re: How to track for 3D model
                Iggorr25 Level 1

                Woow this is one fantastic reply THX a lot, I have to go trough your answer a few times just to understand what you said, but Now I have a good starting point THX a lot

                • 5. Re: How to track for 3D model
                  Iggorr25 Level 1

                  Ok so after a lot of research and watching a lot of tutorials I have found what I need to make the video I want The tutorial that I have linked here will allow me to do the tracking I need and I need to change some stuff in my original; concept of the video but I will be able to pull it of Will post it here after it is done      3D head tracking after effects tutorial - YouTube