My first guess is improper technique and failing to study up on how to use Rotobrush. I'm suspecting you did not freeze the Roto when you're finished.
Use the search help field to find help files and tutorials on using Roto brush. You will find it at the top right corner of AE.
Thanks Rick Gerard, I was trying Roto Brush for the first time and didn't know it required FREEZE in the end to apply rotoscope to the footage.
It is much better now and the render does not look much different from the in-ae preview. But there are still some issues in the whole Roto Brush Tool, for example if Un-Freeze and change some values like Feather or Reduce Chatter, it messes the whole rotoscope and I have to do it all over again or use the last saved file to resume the previous work.
Anyways I will keep looking at adobe Help site to learn more about this tool.
just to make things clear - you do not have to freeze. it's a recommendation because rotobrush is a resource hog and will take a lot of memory. it's not a must but more of a good practice.
as for feather and reduce chatter - these settings are applied after the propagation process (the only thing that is freezed) so what ever happened with your roto brush matte is not specifically related to freezing or un-freezing.
read all the articles and practice. it is quite long but it will save you time and frustration.