It is possible to use channel information to extract a matte. This information comes from color or luminance values. This is called a procedural matte. Usually you would duplicate your footage layer and start applying color effects to isolate certain areas of color or luminance, and then you use that footage as a luminance matte for the layer below.
That said, while it is fairly easy to find edge detail (Find Edges effect) it is extremely difficult to figure out which of those edges should be inside or outside the desired mask area. Trying to deconstruct the auto focus feature a camera uses from the image data would be quite a daunting task and I'm not sure how effective that would be because there is currently no metadata that would tie any part of the frame to something like the face of the person you wanted to use as a mask. Face detection algorithms may have a chance at doing this for faces, but the other parts of the body would be extremely difficult to figure out.
Stereo Photography may offer a solution to extracting a procedural matte from the part of your shot that is in focus providing you took great care setting up proper convergence for the image pairs, but this would not isolate skin tones.
Rotobrush is an automated tool that asks user input to define the inside of the desired mask that uses edge detection and color values. It is very effective if you take care with the original photography, but if you are not careful automatic roto using any tool becomes difficult if not impossible. After the fact visual effects are way more complicated if there is no production planning.
Colorista II is my favorite tool for this kind of color correction. It has a very effective internal matte tool that can do amazing things using color range. Other color correctors (Davinci for example) have other tools to isolate areas based on color or luminance values. Your idea is interesting, but it would be very difficult to pull off using AE's current set of tools.
Perhaps you could use Find Edges and the paintbucket effect to create some stuff on the outside of the subject which could then be reversed, but it wouldn't be a very clean key at all.
I don't think this would work very well. However, your description of your shots sounds like the Roto Brush would do very well with it especially when used with the Refine Matte tweaks
Thanks Rick! Sounds like I'd been a little bit far-fetched.
I've also been using Colorista II and did find it handy. I was just picking up some random ideas from the middle of nowhere, trying to figure out if there were some other way round.
But now my question seems quite settle~
Thanks Szalam! For your answer. Roto-scoping really worked quite well in most cases. I was just wondering if there were another approach.
But anyway you've also been helpful!