It could be a few things. What codec is being used for your source footage? Perhaps try transcoding it to a stable format if it is currently highly compressed. And what kind of file are you trying to create? A QuickTime? If so, what codec? Alternatively, you could try disabling layer cache during rendering, and purging on every (x) frame(s). In order to do that, you have to access the "secret" preferences. Hold down "shift", then access the After Effects preferences. When the preference window pops up, you can release "shift", and there should be a hidden item at the end of the preference list called "secret". There, you can disable the layer cache, and increase the purge frequency. If this helps, don't forget to reset those settings back to their defaults after your render completes. In addition, what keyer are you using? Keylight can hog memory sometimes.
To what drive are you rendering and what settings do you use? In addition to Steve's thoughts, I'm thinking that you may be writing to a FAT formatted drive of some sort, which would limit file sizes to 2GB or 4GB, respectively or at least that the target drive is fragmented or full.
I echo the suggestion that this might be a problem with your drive being formatted as FAT32, which has a 4GB limit on file size. That's the circumstance under which I've seen this error before.
Here's a thought, in case you need a workaround:
Render and export as an image sequence, and then convert that image sequence into your QuickTime movie. There's a tip on this page about post-render actions that tells how to make this essentially a one-step process.