Nobody can tell you anything without any details about your comp, render settings etc.. This could simply be an effect running amok, but thenm again it could be anything else just as well.
You are rendering what is obviously a wrong resolution composition. Too many pixels for Ae to handle. It's in the 20k range and you are close to the 30k limit Ae has. when you are this close, an effect could push you over the memory's edge... where is this video going to play back? This is probably for the web so work with an HD preset HDTV1080.
I see. let me correct what I suggested earlier: you are using the right composition size, but one of your a layers/precomps is reaching an enormous amount of pixels. this could be a result of the layer/precomp size or one of the effects that stretches the pixels (CC Repetile could do that).
1. if this a too big of a layer thing: go over your composition and look for a layer that's very very big. you would see the layer handles stretching forever just zoom down the magnification:
try to see if you can work it without so many pixels. setup your project differently, assemble the big layer out of different small sized layers attached to each other etc.
2. if this is an effect thing: go over the setup and see if you have used and effect the stretches the pixels of the layers and reduce it's values.
Thanks Roei! I just had the same problem and found that I had accidentally had doubled up my Motion Tile effect on one clip.