It sounds like you need to adjust the masks in the nested comps, but you're trying to do it in the main comp. Not good. Nothing happens. Do it in the nested comps.
And if you're doing that, it also sounds like spending some quality time here will be VERY helpful:
It's better to learn AE from the Pros instead of a high school kid with sketchy AE knowledge on YouTube, y'know?
Assuming your masks are in the main comp, this sounds less like a bug and more like a feature that was accidentally enabled. If you need quick support on it, I would suggest contacting Adobe support.
Could you show us some screenshots to demonstrate what you're describing?
If something all of a sudden stopped working then save your project and reboot.
Selecting an editing mask points is always a bit of a challenge but it works better if you use the pen tool (keyboard shortcut "g") And then hold down the alt/option for command/control key to bring up the other tools.