Just use the mask tools. What keying tools are you using? What tutorials are you looking at? If you want to learn about a procedure like keying I would suggest that you type Keying or Keylight in the Search Help field at the top right corner and take at look at the training materials that have been filtered by Adobe. There are a bunch of really lousy tutorials on YouTube and elsewhere.
AE has a big learning curve. I'd suggest you start here: Basic Workflow
A good tutorial on Keylight from the search help field is here: Color keying | Learn After Effects CS6 | Adobe TV
Rick, I appreciate the response. I have After Effects CC. There must be a difference between CC and CS6? I keep following the instructions verbatim on the tutorial and still nothing.
Go to your workspace and select Standard, then select Reset Standard Workspace, then go to View and make sure Show Layer Controls is checked.
Now select the layer with the green screen footage and press the G key to enable the Pen tool. Click anywhere in the image to start drawing a mask. You should see the vertices start to appear. When you close the mask by clicking on the starting vertex you should see the part of the layer inside the mask and the outside of the mask should be black or have a transparency grid if you have turned on that feature.
Masks have worked this way since the pen tool was added to AE. The only thing that could possibly cause problems is user error or your particular version of AE is not up to date and there was a bug. I con't remember any such bug with CC so user error is more likely.
If you cannot draw a mask on any layer let us know. In the tutorial the mask is called the garbage matte. BTW, there's a lot easier way to do the hold out mask than is shown in the video. Even the tutorials on Adobe TV are not perfect, but if you follow the steps I have outlined you should be fine.
Rick, I tried your instructions and it worked perfectly. Thank you. I really appreciate it. I would like to do the same thing in Premiere Pro. Do I simply follow the same instructions? Or is there different way to do this in Premier Pro?
Again, thanks for your help