the first thing I would do is reset my Ae's preferences. here's how: Reset After Effects Preferences
if this still happens, provide some more information:
1. exact Ae version and Os version
2. did this always happen? does it happen on a certain project or setup?
Sorry to see you had trouble with masking. Did Roei's advice assist you at all? Please let us know what's happening.