Well, as you found out: Working with masks in teh 3D view is not the best of ideas. That's one of those oddities thart AE has had forever... Draw your masks in the layer panel instead.
Are you using rotobezier?
No not using roto.
It only happens if zoomed in really far in 3d view.
When I double click my layer it doesnt open in the layer panel. Other ones do but not the one I want?? Its just a simple ai.
Though this thread is a year old, I found it while troubleshooting, so adding to it in case it helps others.
I've run into the same issue as Arrakis, and I just figured it out: I had the layer I was drawing the mask on parented to another layer that had been scaled up, so the effective scale value of the layer I was trying to mask was very small (but scaled up from the parenting). Clearly, AE is treating my layer as if it's 2% of full size, so all pen clicks past the second one are switched to the add-vertex tool, despite the fact that the image is full size on screen.
I'm sure the issue came up in the OP's 3D scenario because to have the the camera zoomed in really far, the layer in question would also need to be scaled really small.
This has happened to me a few times in the past, and I was always mystified, so in case this is going on for anyone else, that's the problem. For me, the fix was to unparent, draw the mask, then parent again. For 3D, you'd maybe want to mask first, then scale it down.
Thank you @Jesse Kerman! I've had a similar issue with the pen tool multiple times in the past and had no idea why it was doing this. In my current situation I was just using 2D layers, but the layer I was masking was parented to a layer which had a scale of 6000% or something, so the scale of my masked layer was only 1.8%. Whenever I'd try to draw a second mask, it would always show the plus/add vertex tool. Not until I zoomed out as far as possible and put the cursor as far away as possible from the original mask did it switch to the regular pen tool. Unparenting worked for me.