Sky Replacement

[Mark_Kolodziej]

Hi After Effects Gurus,

I have a clip from a short musical I was a part of that needs some help! Our VFX artist flaked out on us and now the work falls onto the less experienced...

Sky Replacement VFX Help - YouTube

This shot needs to be corrected to sunset and the super bright sky is making simple color correction difficult. We were also hoping to replace the current sky with a more interesting one. The problem is the footage is so shaky and there are no trackable points in frame the whole time, making camera tracking (at least to my understanding) super difficult, if not impossible. Any advice or is rotoscoping out the foreground frame by frame then correcting new sky position frame by frame the only way to go?

Cheers!

[Mylenium]

No dice. This will throw off even advanced tracking tools. the camera is simply too close in and the details are partially out of focus. You could possibly generate a simple matte with some channel operations and a bit of manual masking, though, so not all is lost. Thankfully the foreground doesn't use any of the background colors and the sky is really blown out to almost white. Start by having a look at the Channel Combiner. A RGB --> Luminosity conversion might enlighten and enawe you.

Mylenium

[Roland Kahlenberg]

Mocha will be able to track the dark horizon in the background. However, you'll need to do some roto work to get around the edges of the horizon since there is foreground movement going over the background's horizon but tracking should help quite a lot.

Luminance matte extraction techniques or the Effect > Keying > Extract effect should help a lot but I'm not certain on the forthcoming results around the fringes of her hair. You'll want different settings for her right and left sides since her right side has much finer hair.

[Rick Gerard · Accepted solution]

I messed with a frame of your video for about 15 minutes and decided that it would probably take me about 4 or 5 hours to pull a decent matte from the footage. Once the matte is successfully created the problem of replacing the sky comes up. Just applying a color effect and overlay would be the easiest because getting an accurate track from that shot would probably take another hour.

With that time estimate, I would say that given a day of fiddling around I could probably kind of fix the shot so that it would look more like it was shot at sunset. It is highly unlikely that someone with limited experience could pull off this correction without a huge amount of frustration. I will tell you that blend modes and Extract are the starting point for pulling a decent matte. It will involve a lot of duplicates and a lot of messing with edge corrections.

If it were at all possible to reshoot that scene you would be a lot closer to getting the look you want. If that is not an option then stylizing the sequence in some other way may be a better option. Maybe turning it black and white.

[Warren Heaton]

You could use the shot as it's own luma matte, but the specular highlights on the actor's face would likely need to be painted or rotoscoped.  How important is the brush immediately behind the actor and the body of water off in the distance?

[Mark_Kolodziej]

The brush and water aren't important at all, it would be beneficial if they were replaced with something prettier haha!

[BartonGarrett256]

At the risk of stating the obvious how about cutting to stills or video of whatever she is singing about, or something else, or shooting her a second time wearing the same dress singing the same song from a different camera perspective and pulling into her face, or cut to whomever she is singing to, without new sound.  Sometimes you can't fix what you have and you have no choice but to improvise.  The true measure of a craftsman is their ability to hide mistakes, everyone makes them.

[Mark_Kolodziej]

Thanks everyone for all the tips and advice! We will be reshooting this or potentially use a different clip....will be less frustrating and now we have a better idea of what we actually need (hindsight is always 20/20).

Lots of great tips though which we have already implemented on some other shots!