Turn off Fast Previes and set the comp to final Qulaity to enforce software mode...
That's basically how I've worked with AE all along. Still freezes I'm afraid.
Nobody knows what this is about?
I too suffered from this error after I upgraded my workstation to HP820z/nvidia gtx690 (driver 311.06)/win7 64. I should note my previous setup did not have this problem (dual xeon/nvidia quadro 1700.) Mask feathering was either excruciatingly slow or would outright crash the software under CS6. I believe (unfortunately for you) that the 11.0.2 update fixed it (it's still slow, though). Once I upgraded to CC, however, the problem seems to have gone away...although pushing feathered mask points seems still slower than it should be.
Not much of a help, I know - but letting you know you're not alone