Reorder your mask and use the Intersect mode. Simply keep trying. The mask modes logic is sometimes odd - when it should be add, it's often subtract and vice versa. Otherwise pre-compose the main mask and apply your otehr mask on the resulting pre-comp layer.
a screenshot would help a lot here for us to give you the right workflow and suggestions. please post it.
if you are planning to do some rotoscoping you should read the following links thoroughly first. this is important and even vital to create convincing composite because 90% good job won't cut it with roto work.
for mask roto:
for other method of rotoscoping you should consider RotoBrush:
In general about mask roto, a good practice would be sometimes to set and adjust you mask in the layer panel and not the composition panel. you set both open side by side and this way you can have it both ways - see the composite, and adjust your mask comfortably in a separate window. you can utilize the various layer view modes. these are similar to refine edge in Ps (now found in the "select and mask" taskspace). you can also uncheck the render box to see your image and mask it too without the mask being rendered. another advantage for this method is that you won't accidentally move your layer.
I tried clicking subtract next to mask two
yes. this should work. I can't really see you have two masks on the same image here.
make sure you follow these steps because looking on your screenshots, I can't be sure:
- if mask1 is Add - it means it will cut your image from you background
- if mask 2 is Subtract it will subtract from the mask above it
- both masks need to be in the same layer
- both masks need to be closed
- mask order is important
Darken/Lighten are for intersecting masks so no use for you here.
BTW I am not sure why you are masking a still image so carefully in Ae, or is this auto trace? anyways, you should know that if it's a still image, and if the path is not supposed to animate, you have much better tools for creating selections of stills in Photoshop. Also, this will save you the render time for calculating the mask in Ae.