Have you checked out all the resources linked here (including in the comments)?
Rotobrush isn't the most intuitive tool in the world so reading up on it and watching tutorials can help a lot.
Have you tried putting color correction effects on it to raise the contrast?
Your question doesn't have enough detail for us to help. Screenshots help. Telling us what you have already tried helps.
But I'll venture a guess.
If your object is moving relatively quickly, then the region that the Roto Brush effect is searching to try to locate the object on the next frame isn't large enough. Increasing the size of the search region can help.
Roto Brush is not a tool for beginners, and it is not something intuitive that you can use without carefully reading through the instructions. Really. Trust me on that. Of all of the things in After Effects, I think that Roto Brush is the one with the largest gap between how easy people think it is and how easy it actually is. Please read the instructions, watch the tutorials, and then read the instructions again as you hit snags.
I did try contrast including color correction and it only last for a few frames and dissappears. Are there any roto brush settings that are relevant to this issue?