What, exactly, are you trying to do? What version of AE? What format/codec is the footage? Can you describe the way your comp is structured?
Edit: and I see this post was from over a year ago. Oh well, at least now they know somebody cares...
Masks have a limited maximum number of anchor points. There's nothing to fix because nothing is broken. Your approach is simply unsuitable. There are too many fine details that eat up all those precious points.