this is the "clean" footage
and this is what happens when i apply an elliptical alpha matte on it
it happens with any kind of alpha/luma matte. made of a masked solid or a footage
i never had this problem with the ae7.0 version, and i tried to find a solution in every common problems or faq thread i found... what can it be?
i tried to activate and deactivate the opengl plugin and the memory settings.
edit -> purge all dosen't solve the problem, neither switching to the single monitor
notice that if i use a mask on the layer instead of the track matte it works good
video card en8800gt 1gb
Perhaps an imbalance in the values stored in the matte layer which happens when processing LDR footage (8bpc/16bpc) in 32bpc mode...? The usual fix would be to pre-compose the matte or apply a HDR compander effect, if other matte modes don't resolve the issue. You might also try ther Set Matte effect, as it will enforce a different processing mode. Should not happen, though. Solids and masks should be 32bpc safe.
Also check whether your footage interpretation for the layer you intend to mask out are correct. If it has set an Alpha channel that is fully opaque and thus has no further use, turn it off as it may cause similar issues under some circumstances, due to the values in the channel not being uniform at all.