That would require some form of contrast for an automated analysis, but since nothing is there (or not enough at least), that is bound to fail, so using smart brushing or masking techniques is probably as good as it gets. The only other alternative is obviously a different lighting setup and a different color correction approach, but you very much ruled out that one, so I'm not sure what else to suggest.
This guy simply copied & pasted a query he made on the Creative COW:
Apparently he needed to hear from another source that what he wants to do isn't practical, nor was he very forthcoming with details of his effects shot that would let people offer alternatives.
To get proper guidance/solution for your problem let us atleast have SCREEN SHOT of the shot first
I'm sorry to offend or hurt your ego, but contrary to your belief, your opinion IS NOT the be-all-end-all in the industry. There are second opinions (many of which are provided by YOUR peers on both forums) and I would like to hear them...if that's all right with you.
I mean, I know that as soon as you say something I should just give up any and all attempts at researching alternate methods, as your's is the last and final word on the matter. How silly of me to even waste my time pursing the intelligent opinions of people not working for KCRG.
Anyway, to the other folks who lent helpful advice on methods I could try manually, I will give it a shot and let you know how it turns out. Thanks for the info!
Lighting is dimensional. It requires geometry to look real. You could roto a mask for one side of the face, apply a gradient to simulate fall off or use a combination of gradient and a color chanel to control an exposure effect. It wouldn't be a simple or quick solution but it would kind of simulate 3 point lighting.
If you put up a screenshot this weekend I'll try and send you a comp that shows one option.
Have you seen the extras for the original Tron? Many of the light effects were roto'd frame-by-frame. (Like the shots were the underside of Tron's disc projects light onto Tron's face and body) The upside was that the images were grayscale and the light was always moving and the sequences were short.
But even the best hand roto'd shadows always look a little off. I don't know what your shot is, but if it's close-up, lingering, and in color, it's only going to make it that much more scrutinized.
The only other thing I could think of would do be to use a 3D proxy model of your subject as a shadow catcher and recreate the lighting setup in a 3D app. Something like PFTrack 5 can automate this through geometry tracking. But since it sounds like the budget isn't limitless, you'll probably just want to hand match the model's rotation and translation to match your subjects.
But really really, you should just reshoot it. No insulting your AE skills, but even a VFX whiz could spend days in post trying to create a fix that's going to end up looking crummy anyway.
" If you put up a screenshot this weekend I'll try and send you a comp that shows one option. "
I have not put the original question
Without the screen shot of the actual situation, or what is the subject matter,
It appears that we are all trying to solve the mystry
Apparently there's no screenshot so there can be no solution.
I'm glad to know you're familiar with KCRG.
As others in this thread have indicated, you aren't providing sufficient details about the nature of your shot, which would allow them to offer helpful suggestions. No screen shot, not even a detailed, written description which could give people a chance to ask clarifying questions.
If you're in the position where you can't share the secret, I'm afraid helpful replies must also remain a secret.