Without actually seeing your settings nobody can tell you much regarding the mask issue. Have you tried changing the mask order in teh timeline? Are the masks properly closed? What else? System info? Version of AE? Footage types? Comp settings? The paint issue can't be resolved any other way. That's simply AE's age-old paint engine and its limitations.
Fair enough. I'm using AE CS6 here, OS 10.8.5.
I'm very familiar with paths, masks and retouching, very unfamiliar with AE.
The AE acrobat help file recommends making masks on a solid white layer with the video turned off, so you can lock the actual layer being masked. I see the rationale for this but don't see how to apply the masks from one layer to a different layer.
I've got two masks working correctly together on the white layer now; how do I mask the layer I actually want to mask? Current layer order, top to bottom is white, BG extension, shot video.
It would be really helpful to see a frame of the original video. From your description all I would do is put the set extension on the bottom layer, the video of the dancers on the top layer and do a rough roto of the top layer that keeps the dancers and reveals the set extension. Maybe I'm missing something here but I think you are way over thinking this. Imagine that the bottom layer matches the curtains and the stage and the two groups of people are your two dancers:
I would type Masking in the Search Help field at the top right corner of AE and study up.
Yes, I hear you. I have layered things that way in the past. My thinking for this one is that I'll need to be painting shadows (and blurred feet/limbs) on the extension layer here and wanted to do everything on one layer.
The solid white layer with video off is still attractive as a way to avoid moving the image layer. I'll look further as you suggest first.
ah, never mind.
simply copying the mask from the white layer and pasting it onto the BG extension layer seems to be the approach, yes?