Have you opened the Properties panel for the clip you've applied Chroma Key or Videomerge to, set the key color and adjusted the color range?
Let me tell you what I did so you can know if I understood you. In edit tab, I click "Edit Effects", select "Video merge" drop menu. And when I hit select color and then select the color the mask works perfectly. The subject is a boy playing guitar, background disappears perfectly, but I can see through his shirt and hair. I have tried adjusting the tolerance (no change) and I adjusted the presets (no change in opacity but I like the soft edges for this project).
If I unselect the "Select Color" toggle, it masks perfectly, I assume the program just picks the right and most obvious color.
As I say in my book, Videomerge usually works very well with a perfectly shot, perfectly lit green screen.
If it doesn't work, you may need to work with one of the other key effects, like Green Screen Key, Blue Screen Key or Chroma Key.
And, if there is an uneveness to your shot or you have ragged edges, you may need to combine effects.
As Steve says, there is more to doing Keying, than to just have a green background. The lighting is very, very important, as is the choice of the green. This ARTICLE links to a site with some Green Screen tips.
I don't think I'm explining the problem well enough.
Here is a clip on YouTube that should show the problem.
check out the four girls singing at :12(notice the gilr in a blue shirt, you can see through to the back video), the boy playing guitar at 1:13 (notice his shirt and hair), and then after the main part of the video I added a test shot, at 2:42...(there's a big black blank spot between the end of the real video 2:20 and the begining of the test at 2:42) in the test shot it works perfectly.
all I did to the video merge that shows through is crop them, scale them, added some motion to the guitar player. the opacity level is set to 100%.
These are exactly the issues Bill and I have been addressing, Obediah.
Videomerge may not have enough power to handle these clips as they are shot.
I just don't understand why it would work in one video merge effect and not another. It seems that if my lighting was off, or the background green was off it would always have bleed through in editing. But it works fine once (the final shot of that sample) and than not.