what setting is not correct?
you should first get familiar with Rotobrush before starting to use it since it's not intuitive (like most of Ae). type rotobrush in the search field at the upper right corner of your Ae interface and get cracking.
if you still have questions, share some information about your software (version number), Os (same), setup, footage, composition - show some screenshots and specify the steps you take so we can better assist you.
The only time I've encountered something that seems like what you're describing is when I've forgotten to enable keyframing on my Mask Path prior to drawing with the Roto Brush. Perhaps that's what's happening?
How terribly frustrating. Are you still having trouble with rotobrush? Did the experts' advice here help at all? Please let us know if it did or if you still need help.