Here are some things to consider when planning to add elements to hand held footage. I'll list them in the order that I consider things when I plan VFX shots. I've been doing this kind of thing for more than 40 years and this works pretty good for me.
- Plan the camera movement so that there is plenty of fixed geometry in the scene to track (you can't track anything in a shot of the ocean with the surf rushing in or a shot of a sidewalk full of people shot with a telephoto lens)
- Keep the camera as steady as possible
- Shoot at a higher than standard frame rate if it is practical to get more information to track (59.94 preferable over 29.97 which is way more preferable than 24 or 23.976)
- If your camera has rolling shutter problems and most consumer and prosumer cameras do, do everything you can to minimize them
- Manage lens distortion by using lenses or focal lengths that do not distort the edges of your frame
- Avoid jerky camera motion that introduces motion blur
Follow these six rules for camera usage and you will have a fairly good chance of successfully pulling off accurate tracking.
Now let's talk about composition. This is more about concepts than a list of rules.
First concept: If you have a moving camera and you want to insert flat elements into the scene by attaching them to some object in the scene try and get that object fairly close to square to the camera at least once in the shot. For example, you have a picture on a wall that the actor walks past and you want to replace the picture. You should frame that shot so you get a full view of the picture at some time in the shot and that it is pretty close to square to the camera. You can correct for perspective using tools like corner pin but if the painting is at a 60º angle to the camera and greater it's going to be hard to get a good track. it is important that you pay attention to the perspective in the scene and how that perspective changes as the camera moves.
Second concept: If you are going to have to separate something from the background, like your actor walking around, you must be aware of the background behind the actor. The simpler it is and the more it is different than the actor's skin, hair and clothing color the easier it will be to separate the actor from the background. The techniques you will use are Rotoscope or procedural mattes generated from color channels (keying).
If you plan to insert 3D objects in the scene that interact with the actor or the set you must place enough fixed geometry elements in the scene for the Camera Tracker to calculate the chance in perspective and parallax. If your shot frames the actor in 70% of the frame and he is walking around a neighborhood with parked cars with reflections in the windows, traffic moving on the street and other folks on the sidewalk moving in all directions it is going to be very difficult to find enough detail in the shot to get a good camera track. The simpler the shot the better for camera tracking.
i could go on and write several chapters for a book on how to make and set up these kinds of effects shots but this will get you started.
When you get your footage into post you'll need to first make a rough edit of the footage so that you are not spending any time working on footage you are not going to use. A few frames on either end of the shot (handles) are OK but tracking a 2 minute hand held shot where you actor walks a half a city block is a fools errand. If there are 200 frames in the middle of that show where you want your actor to walk past something you put in the scene then you trim the shot to those 200 frames and work on them, then cut them back into the original shot.
Once you get a shot trimmed you have to figure out which kind of tracking will work the best for things you want to insert. If you are inserting flat elements like replacing a sign then corner pin tracking in AE or Mocha will probably be the most efficient. If you are going to do a lot of rotoscoping too separate the actor from the background so you can insert something behind him then you might want to Stabilize (not Warp Stabilize, that's another thing entirely) the shot. Stabilizing means that you lock some part of the shot into position so it does not move to make it easier to work on. Then you do the roto, add the elements, then remove the stabilizing from he shot and apply it to the effects and roto work.
If you want to insert a 3D object into the scene then you need to Camera Track the shot. I have done several shots shots where I stabilize the shot to do the roto, put the motion motion back in the shot, then camera track and add my 3D elements into the shot and use the roto as a track matte for the 3D element.
There are a fair amount of tutorials on the web that show how to use these techniques. If you type track or stabilize or camera track or track in mocha in the search help field at the top right corner of AE you'll be directed to a fair amount of community resources that will help you get started. Without actually seeing the shot you are going to take it's hard to point you exactly in the right direction. If I was teaching these techniques in a class I would expect to spend at least a couple of days of class time going over how to set up and shoot the shot then the rest of the week would be spent on the basics of tracking and roto and masking. Then a day or so on color correction, adding shadows and the rest of the things you need to do to sell this kind of VFX shot.
If this were my project I would budget about 2 hours for each 20 seconds footage where the actor passed in front of a CGI element.
The best way to get started is to just shoot a couple of shots with your phone and try some of the different tracking techniques. Try just inserting a simple text layer or solid. For example, here's a very quick test shot showing how to add some text to a wall using the Camera Tracker and use a 3D solid as a track matte for the text. Sorry there's no audio.
Here's another one adding a puppet from C4D to a Camera Tracked scene.
Both shot with my phone to use as a very quick demo for a class I was teaching.
Thank you so much for this instructive and helpful introduction. Greatly appreciated.
I will spend some time to digest this and follow your suggestion of doing a few tests with iPhone shots.
Many thanks and peace.