Without seeing the footage it's really hard to say what is going on. You rotobrush in one direction only, you pause when the matte wanders off and make corrections on the frame where it goes bad. Once you are done you transfer to the refine edge tool and start from the beginning moving in the same direction that you moved before. As soon as you are done, especially if the clip is longer than a couple of seconds, you freeze the roto and then check it. If there are any problems go back to where the matte fails, unfreeze, and start in the same direction you were going in (I usually go forward in time) and then you have to go clear to the end and freeze again.
Rotobrush is NOT intended for shots that are a couple of minutes or longer. You'll run out of system resources. It is a good tool for some masking jobs, but an actor standing in front of a white wall may be a better candidate for a procedural matte of some kind rather than rotobrush. I would have to see the actual footage before making a determination.
If you are new to Rotobrush then I strongly suggest that you go through some of the training materials you will find if you type Rotobrush in the search help field at the top right corner of AE. I've been using AE for a little more than 20 years and I had to go through a bunch of tutorials before I could figure out how to use the tool properly.
I hope this helps. Show is the shot and give us some more production details if you need more help.
Thanks for the reply, the clip is only about 7 seconds but I may have accidentally gone backwards which could have triggered the brush to get confused. I'll have to try it once more as it created a fantastic key from the frames I saw, and if not I might try other methods. Thanks again!
You also might want to consider rendering a Digital Intermediate using a nice production codec (sometimes called a mezzanine codec) as soon as you get the Rotobrush done. Rotobrush is terribly system intensive and bloats your project like crazy. It can also easily foul up. I always do my Rotobrush work with only the footage layer in a comp and then, if the shot is longer than 2 or 3 seconds, I always render a DI and then just dump the Rotobrush comp from the project. I never use a Rotobrush layer in a comp with other footage and then try and apply effects. The closest I would ever come to that would be to use a nested short Rotobrushed comp in another comp.