It's really hard to help when the question is so broad and vague. If you post screenshots and explain exactly what isn't working so well, then maybe we can help.
One thing to keep in mind is that Roto Brush relies on differences in pixel values to do its segmentation. If there aren't clear differences in the footage, than it is going to be hard.
As with all sorts of things, you may have better luck cranking the contrast way up for the copy of the layer on which you're creating the matte, to accentuate the differences.
If you are dealing with black and white footage you might want to dive into some channel operations....
There are many places where you can find good channel ops tutorials, but take a look here (the ones about masking specifically)
This is a Photoshop based tutorial page, but the concepts apply to After Effects as well. You won't be able to use doge and burn tools, but with a combination of garbage masks and channel ops, you should be able to come close
These should help a lot. I figured that there was another way beyond the obvious contrast tweak. I had seen it done with color channel isolation before but not B & W.