This may be an off-the-wall idea, but I wonder if it might be the kind of footage you're rotobrushing. What's the codec? What's the media container?
The footag e Apple prores 444 24p from Arri Alexa. Can't seem to find a way to avoid the error.
Can an Alexa add 3:2 pulldown using certain settings? I don't remember. If it can add pulldown, I'd check the footage and remove it prior to using Rotobrush. Make sure it's at the proper frame rate (23.976 vs. 24), and that the comp's frame rate matches the footage frame rate.