So I've been giving this footage to key out in After Effects as a test. In theory I can use any other Adobe software that I want to help out, but I think it looks a lot better if it I achieve my goal purely with AE. I've got the entire Adobe CS5 Master Suite, by the by.
Now I'm using Keylight 1.2, have read through the entire PDF manual and am also using Simple Choker....but no matter what I do, the results around the edges (partiuclarly the hair) are just terrible. I'm inclined to blame the footage itself since I don't think it's been lit well and the subject is way too close to the greenscreen so that there's more spill then can be compensated for - but maybe I'm missing something? I'm reluctant to tell the person that their footage is crap as they're a potential employer and my getting a steady gig is on the line. Please help!
A Few Things to Know
My current Keylight Settings
My current result (with lots of flickering edges, vanishing and reappearing hair on the left side and...hoo boy)
And yeah, it's a bit silly. My full name is already crazy easy to find online so I can only hope I'm not sabotaging my life by making it available here!
What Happens if I try Despill Bias (linked to Alpha Bias), which is what the manual suggests
The most important thing is to not try to do the entire shot with only one matte.
You've got a garbage matte, and that's good. Now, make a core matte that preserves all of the model but the edges. You can make this core matte by using the Keylight effect and then choking the result.
Then, just use the Keylight effect on the thin region between the inside of your garbage matte and the outside of your core matte. That's where you need to fiddle with the details of the Keylight effect.
In fact, you might even cut the edge matte work into two pieces, one for the easy work of the lower frame (the shoulders) and one for the hard part: the hair.
Also, you can hide a lot of bad keying work by having the new background be similar to the hair and greenscreen color.
Finally, don't forget the Refine Matte effect, which can help a lot with matte edges in motion.
Oh, and you should do yourself a favor and buy this book:
The chapter on keying describes how to do what I just outlined, and it does so in clear and comprehensive detail. (Full disclosure: I'm the book's editor, so I might be biased.)
I saw a four part tutorial yesterday that deals with the topic of multiple masks and creating core/garbage mattes - and also might help you with the hair issue you are having. Go to the following link, and search for "Multi". as I said, there are four parts. I hope they help.