Miracle Camp wrote:
I would even start using the pen tool to mask it out at this point, but who in the world would want to do that?
Who would? Lots of folks would. Why? Because that's what everyone did before the Roto Brush and it gets better results (not counting the results one can get with the Refine Matte tool). Granted, for some shots the Roto Brush is faster. However there are things you can do to speed the pen tool process up. (Sometimes it's even quicker than the Roto Brush.) Here are a bunch of tips and tricks for rotoscoping.
That should help you out in case you can't ever get the Roto Brush to work.
If, however, you really want to use the Roto Brush, folks here are going to need more info from you to start troubleshooting. What (exact) version number of AE are you using? 11.0.1? 12.0.0? What format AND codec is your footage? What are your computer's specs? Etc.
Rotobrush needs detail and contrast and edges to work. If you are getting the entire background selected then there is not enough of any of that in the background for the tool to work properly. I don't know for sure because you didn't provide a sample of your footage as a movie or a screenshot.
You may also be going about things backwards. You may be trying to select the background when you should be trying to select the foreground. Again I don't know because I have not seen the footage.
It's also very important to realize that all roto takes time and work. Rotobrush speeds up the process a lot for certain shots, a little for others, and not at all for even others. When you plan your roto work you should only be doing the roto work on the shot from the first frame where something needs to pass behind the rotoed object until the last frame where the object emerges. Most of my shots requiring roto work are cut up or split in the timeline so I am only working on a few frames here and there and only concentrating on the edges where the object and the rotoed layer intersect. Everything else is a waste of time. Here's a perfect example where I had a small animated character that needed to pass behind a chair. The character was shorter than the chair and was only behind it for a few frames.
When I gave this footage and assignment to a class a while ago most students did the hand roto work to the entire shot and carefully animated the mask around the entire chair for the entire panning shot spending 20 minutes instead of 5. Check the info palette. The only accurate and careful masking required for this shot was the far right edge of the chair for 17 frames.
Take a few seconds and send us a screenshot of your footage, explain any movement in the frame, and we can give you some pointers.
Here are two screenshots. They are both dolly moves to the right or left, which I'm sure is hurting the rotoscope too.
I didn't mean to trivialize the pen tool and building masks, of course, but I'm doing this project as favor, and I don't really have the time to do so. I'd rather rely on the rotobrush if possible.
I'm running After Effects CC and I'm up to date.
I've done other rotobrush jobs in the past, and I understand they take a lot of time. The fact that I can't see a preview of what I am rotobrushing is what is making this project so difficult.
My friend shot on a Canon 6d at both 1080p 24fps, and 720p at 60fps.
What are you trying to isolate? The guy on the lawnmower should be fairly straight forward. The trees in the foreground? How about the girl in the red shirt? Her hair is going to be tough but doable in CC with the new tools.
In rotobrush you can see the transparency by selecting different preview options. You just can't see the composite. Here's a quick look at rotobrush on my sample footage using a green overlay to see the edges better. Green gave me better separation than red. That's how I work.