If you can make a screendump of this mask on the timeline we can see if we can reproduce?
Mac or windows?
Sorry cannot reproduce either in CS6 or CC.
Works like expected.
Have you tried with GPU set to software?
Ahh there we go!
Mercury Playback Engine in hardware mode ignores blending modes during transition. In software mode it renders fine.
I think some issues are system related.
For instance I have an issue with directional blur.
In software mode its fine, in hardware mode it shows through on the edges.
Others haven reported to work as expected...