You cannot track masks in CS6 or before. This (limited) feature was introduced in CC only. However, nothing stops you from creating the mask on a separate layer and play around with matte modes and applying the tracking data to that layer.
I would approach your problem using a track matte instead of a mask and tracking is the way to get the mask to stick to the hand held shot. Depending on the motion in the shot you could use Mocha or Track Motion or even Stabilize Motion to accomplish the task. If you use Stabilize motion it's really easy to put the motion back in the shot and add that motion to the track matte. I might go with that method because the top of the window that you are trying to punch a hole in has some really good tracking points. If the shot moves a lot and part of the window goes off screen then you would use Mocha. Let me make assumptions that the shot basically keeps the window in the frame and Stabilize Motion or Motion Tracking is the best solution. Here's the workflow.
- Open the Motion Tracking Workspace
- Add a new solid to the composition and name it "matte' - try orange and set the blend mode to multiply so you can see through the layer
- Open the footage in the Layer Panel and make sure it is selected as the tracking source
- To Motion Track the footage pick two really good detail areas to track like these
- Track including position, scale and rotation to take into account all of the camera movement
- Make sure you have a good track and then set the target to the matte layer and Apply
- Go to the first frame and draw your mask on the matte layer using the pen tool
- Set the matte layer as an Inverted Alpha track matte for the footage
That's all there is to it. You're done -- With the simple solution.
As I thought more about your problem it occurs to me that the thing you see through the hole in the window is going to move so if you want to add a new background that is visible through the hole you're going to need to add the camera movement to that layer and to make it really work you are going to have to do something about the parallax shift between the foreground and background. Your best option for the best result would be to use Stabilize Motiion instead of Track Motion.
Here are the steps:
- Go to the Motion Tracking Workspace
- Rename - this is important if you are going to use the animation preset I will give you.the footage layer 'stabilized'
- Select the 'stabilized' layer as the Motion Source to set up Stabilize Motion and check scale and rotation
- Select two good areas with detail to track as in the first example
- Run the tracker, verify the track and Apply to stabilize the 'stabilized' footage
(this will give you a stabilized window that you can easily work on)
- Add a colored solid to the comp to act as a track matte to cut the hole in the window
- Draw the mask using the pen tool on the solid to match the hole in the glass
- Set the mask layer as an Inverse Alpha track matte for the stabilized layer
- Add the new background layer - the thing you will see through the hole in the window to the timeline below the window layer
- Add a new null to the composition and apply this animation preset to the null : Dropbox - destabilize Rotation Scale.ffx
(The animation preset will add expressions to the null's position, scale and rotation that read the anchor point, scale, and rotation of the 'stabilized' layer)
- Move to the first frame of the timeline and Parent the new background, the stabilized footage and the matte layer to the null
(this will remove the stabilization from the footage and add the movement to the matte and the background)
- If this is good enough then you are done - but to add paralax to the background make all layers 3D then move the replacement background layer back in Z space to put some distance between the background and the window
Here's why the null works to remove the motion from the stabilized layer. When you Stabilize Motion and include rotation and scale keyframes are added to the anchor point, scale and rotation of the layer that remove the motion. A very simple expression that connects the position of the null to the anchor point of the footage. A little more complicated expression connects the scale of the stabilized footage to the scale of the null but it calculates the difference between the two scale values and subtracts that value from the original scale of the null. The rotation expression simply uses a minus sign - to reverse the rotation. When the null is parented to the original footage all of the anchor point, scale and rotation values are counteracted. Parenting the other layers to the null puts the camera motion, scale and rotation in those layers.
When you convert all of the layers to 3D then the footage and the matte don't move around because they are at the comp center but if you move the background layer away from the center of the comp and scale it up so it's the right size the difference between the movement of the layers will give you the same perspective shift and parallax effect you wold get in real life.
Hope this helps. Compositing almost always involves a lot of thinking and a bunch of layers.