Have you tried garbage mattes? Have you tried multiple instances of Keylight? All we need to see is one frame. Just grab a screenshot of your project WO effects.
It's reasonable for us to ask more about your footage. DV footage is very difficult to key and requires very different techniques than HD lossless 10 bit footage. Post a frame, one where you're having difficulty, tell us about the footage, and we can help.
BTW, you haven't tried everything until you've found the solution.
As Rick said, provide screenshots of the footage and the keyer settings. I tend to agree - sounds like the classical "I was too lazy to draw a garbage mask" and the screen is not uniformely lit to give the proper result just with the keyer...
@Rick: "sounds like the classical "I was too lazy to draw a garbage mask." Really?
I don't even know garbage matte just yet. I'm still new. You're right about the lighting. It's frustrating of the lighting I have.
@Chris: Sounds good. I'll upload it here in the morning.
Have you tried Premiere Pro's Ultra keyer? It's pretty remarkable at keying poorly lit shots. And it's far less fiddly than Keylight.
I've been really impressed at the speed and quality of it (esp on supported GPUs). It is an order of magnitude faster than Keylight.
Key your footage in Premiere, then use Dynamic Link to pull your PremPro project into AE. Then do your color correction, light wraps, and all the fancy stuff in AE.
I've never mesed with Ultra. Does it work with animated garbage mattes if you need to use them?
Yes, but PPro's garbage mattes are on par with Final Cut 7 – basic and servicable. You have animatable 4-point, 8-point, and 16-point polygon mattes. Nothing elegant like AE's bezier masks.