I'd suggest searcing for some keylight tutorials. You can't just click and go with keying. Sometimes it requires garbage mattes, inner masks, carefully adjusting and even animating the controls. We can't give you any better advice unless we know more about your project and see a frame or two.
In addittion to what's already been mentioned, what exact version number of AE do you have? If you're using 11.0.0 or 11.0.1, you are missing some bug fix updates.
Also, some more specs on the footage would be useful (frame rate, format, codec, etc.) as would information regarding your system specs and some screenshots demonstrating the problem.