You've referred to masks a couple of times here. Did you actually create masks in some way, or are you just referring to the matte created by the Roto Brush effect?
Tell us more about the characteristics of your composition, source footage, and exported movie---including frame rates. Are you working with interlaced footage? If so, did you separate fields? Did you get a warning when using the Roto Brush tool about frame rates?
Check your composition framerate and your footage interpretation. Seriously sounds like they don't match. also make sure to work at 100% zoom.
I've been working with Apple 422 footage and Roto brush. You have to work at 59.94 if the interlaced footage is 29.97 otherwise Roto brush won't render the mattes with interlacing. I did do a Roto brush at 29.97 and had to render without interlacing, that was the only way to get it to render. After making that mistake I realize the necessity to work in a 59.94 comp, then nest into a 29.97 and it renders fine with interlacing. Hope this helps!
Freezing the roto brush fixed my render.