Interlaced footage with incorrect/unsuitable footage interpretation would be my first guess. And working at half res naturally qualifies as equally bad - by suppressing evey second pixel, you inevitably the result of the analysis is different than it woudl be at full res. in fact that's why you should get a warning about it when applying the RB the first time...
Depending on the shot (if he flaps his arms or not), this should be pretty easy to do with Mocha. The lines are pretty clean and well defined.
Draw a shape for each bicep. Motion track. Draw a shape for the chest. Motion track. Draw a shape for the shoulders. Motion track. Draw more shapes to fill out the remaining shapes and attach to the tracked motion. Adjust as needed. The key to good rotoscoping is not to try to do everything in one shape.
Navarrow is right about roto. whether you use Mocha or just roto by hand in AE, this shot shouldn't give much trouble. If you're set on using the roto brush, you'll probably need to go through the footage a bit and re-establish FG and BG to help the Roto brush find the edges. Remember that the Roto brush relies on contrast to work properly.