You should always use the render queue when possible. Never use the export function unless you need to. Have you tried enabling the "purge" function in the secret prefs? Hold down "shift" when accessing your prefs, and then navigate to the secret section. Disable the layer cache, and enable the purge. You can start with a value of 10, or so, but you may need to set it to purge every frame.
What are your comp/project settings? How big is the comp you are rendering?
After your project renders, don't forget to reset those settings to their defaults (disable check box).
Well, you could simply buy Primatte... ;-) I used to think Keylight was more than okay, but it seems The Foundry have completely lost interest in it and do not fix any meaningful bugs anymore, so errors creep up more and more. As a workaround, you could split the timeline in question into multiple segments. The way the render queue works, this will have AE purge everything before starting a new job. You can always assemble the keyed files later. Of course do not forget the "secret prefs" that also provide options to purge every n-th frame...
Also, make sure you upgrade to the latest version of Keylight for AE:
Is there a better plugin for green screening then Keylight?
Not for the price . . .
Keylight is more than adequate for most keying with most footage. If you've got real problems with your footage you can sometimes pre-process the footage to get a better key. If you're having lots of problems then a few frames (4 or more) of the original footage would help us come up with a solution.
Keylight is used on hundreds of films each year, so it can be considered an industry standard.
If you're having image buffer problems, please follow these instructions:
- Jonas Hummelstrand
I've never had any problems with Keylight running out of memory (and I've done keying on 4Kx2K movies)
Keylight is great. I also really like Primatte 4. It has a friendlier UI and built-in light wrap and digital video deblocking pre-processing.
I have been using After effects for some time now (3 years or so) and I regrettably have to report it has been getting more laggy with each "release". The Keylight error occurred to me too some days ago. I tried a lot of solutions following some tutorials on how to tweak AE in order to get more memory. I haven't had any success. I had to render to Quicktime with PNG output and re-import. AE still lags almost unbearably but at least I don't get the keylight error.
I wonder at this point what kind of computer is needed to run this software. I am hugely disappointed.