I'm trying to understand why this is happening to some layers I've got in a comp in CS6.
I have a layer with some styles applied to it: gradient overlay, stroke, etc. When I set the track matte of that layer to 'Alpha Matte <layer>' the layer refuses to get masked properly. However, when I remove the styles the layer is then masked out as expected. Is it not possible to use Alpha Matte track mattes when a layer has style applied to it?
As Rick said, provide more info. Should not bne a problem, but perhaps you use some odd combination. In particular only partialyl transparent areas can change the appearance of some layer styles notably...
If your matte has soft edges or variable transprency Stroke will foul that up. Everything else should work just fine.