You might want to look up TRACK MATTE options.
You can either use the green layer as a "cookie cutter" for the tropical picture, or even use it with(or as) adjustment layers for different color corrections.
Make sure that the layers are the same duration. The lower (fill) layer is probably longer than the matte layer.
Note: If you're using a still image as the matte layer, make sure it has been imported correctly. It's possible that a photoshop sequence was imported rather than a single frame.
It appears the track matte (layer 1) is the same as the one on the bottom (layer 3). Are you using a keying effect to create the selection within the frame? Is is possible that the effect has a keyframe or rogue value that stops creating the alpha channel at some point?
Alt diagnosis: select Edit / Purge / Purge All and try again. The image may be stuck in the video cache.
Note: reflection on the table is going to be composited too?
I am still very new with after effects so forgive me. I used rotobrush to select the green screen. I found that it was more successful then the keying option. I get confused with the track mattes. This is what layer one looks like before I add layer 3. (check below) So they aren't the same, but they originally were I guess.
Make sure the roto brush extends to all frames of that layer.
I'd probably use the effect KEYLIGHT to achieve the cutout. More difficult to understand, but more accurate results.
link: Keying in After Effects
How do I achieve that... After Effects can be so frustrating. There such simple things, but so hard to achieve. lol
Keying wasn't able to pick up everything as well because of the shadows I have. Also the reflection on the table caused problems. But with the rotobrush I got everything within seconds. That's why I wanted to try it with this.
You are approaching the problem in the wrong way. You are trying to put the video on top of the picture frame when you should be putting the video below the picture frame. Rotobrush and track matte were not good suggestions. Rotobrush is useful for removing certain moving parts of images when masking by hand is too tedious but it does not work for all shots, takes at least a half hour of reading and practice to use correctly because nobody I know ever got rotobrush to behave the first time by just poking at the controls.
We could have pointed you in the right direction quicker if you had been more detailed in your description. Also, screenshots that just slow layers are pretty useless. Here's what you should do:
Put your footage with the picture frame on the top layer.
Put your footage with the tropical scene on the bottom layer.
This shows the Picture Frame with a simple hand drawn mask. The tropical footage is below but is temporarily turned off (click the eye) and the transparency grid is enabled so you can see the hole:
All you have to do is turn on the Tropical footage (bottom layer) and you are done.
Because the picture frame appears to have green on it you could also use Keylight to make the green transparent.
Drawing a mask is really easy if the camera is locked down on the picture frame shot. If not, use keying to poke a hole (alpha channel) in the top layer. If the camera is moving you'll also have to match the movement and apply it to the bottom layer. That's called Motion Tracking and you can learn about it by typing motion tracking in the Search Help field at the top right corner of AE.
If you ever have problems with a project then the best way to get help is to write a detailed description of what you are trying to accomplish, then select the layers that are giving you problems in the timeline and press the U key twice to reveal all modified properties of those layers. Take a screenshot of that whole thing. Then post the whole screenshot. Include precise descriptions of your footage and anything else you did to modify any footage or effect in the comp. That's what I did with the screenshots. The only thing I did to the top layer was add a mask set to the subtract mode. There was nothing to do to the bottom layer.
I'm sorry I didn't get straight to the point, and that I haven't been more detail with my questions. Originally I was trying to keep the text to a minimum so I wouldn't scare off anyone. So much for that... But I have learned my lesson now.
I like rotobrush, and I am able to use it with the help of a tutorial. I just dont know how to apply it. I tried it by hand like you suggested this time. However my first problem has already appeared. How do I get the opposite version of what I have. (check image) I need the background not the foreground. Your screenshot looks nearly identical to mine. And yes I my camera is locked down. Thats why I wanted to do it this way.
Appreciate the help. Been going mental doing this stuff alone.
The brush is completely the wrong cool for your task.
If you followed a good Rotobrush tutorial you would see that there is an invert function for the mask. You absolutely must use Roto brush, which is slow and inefficient for this job, then just click the invert switch on the effect.
If you were going to effectively use any software you need to learn good workflows. To learn good workflow you need to vet the folks that are giving you the training. Make sure that they know what they're talking about. Most of the new tutorials I have seen on YouTube in the past several months have been created by amateurs and propose incredibly any fission and ineffective ways to do things.
One use of the "search and help" field at the top right corner of AE for rotobrush would have directed you to community resources that explain the invert function. One search for "masking" will teach you how to use the masking features.
I am not trying to scold you, just trying to give you good advice that will make you a more effective and productive person.