You have not provided any exact system info or told us what version of AE you are referring to, so this is not of use to anyone. you have to be much more specific.
Are you still struggling with Roto Brush? Please let us know.
I have moved on from that project and haven't been doing much with brush toolw recently. So I opened that project to see what would happen and yes not a minute into using the roto brush it stuck on, as if the click to paint and release to stop painting were reversed.
It may have something to do with this particular project file. I was trying to recreate the problem in a new project and could not get it to happen.
But in this original “problem” project it happens.
The project is a simple 20 second 1920x1080 23.976 composition with two layers of 1920x1080 23.976 ProResHQ video.
iMac (Retina 5K, 27-inch, Late 2015)
4 GHz Intel Core i7
16 GB 1867 MHz DDR3
AMD Radeon R9 M395x 4096 MB
Wacom 851 tablet, or the Apple touch pad, or a standard USB mouse