can't fix the memory issue but I'd suggest you purge all caches and memory and render to an image sequence. If you use Multi Machine output settings it'll skip existing frames and that way when it crashes you at least don't loose your progress and can start rendering from where you left off until your memory fails again. then when you're done you can just stitch the frames together and render w/ audio if needed.
annoying for sure but it's more likely to work than your current setup right now.
Purge your image cache first of all, second, are you using AE to render or AME?
If using AE, render it as a TIF sequence and then assemble the images in another programme, either Premiere or WMM. Or send it to AME.
What amount of RAM have you allocated for AE, and what are your render settings?
I'm using AE to render. I have the max amount of RAM AE will let me allocate (12.9 GB). I purged everything and tried to render out the comp as a Tiff sequence and got the error "Unable to allocate enough memory to render the current frame (2713 x 1719 @ 8 bpc)" about 3/4 of the way thorugh. Before trying tiff I was rendering out to a 1080p 8-bit DNxHR quicktime, video only.
I've done several projects almost identical this and with no issue (I rendered out two comps last week with an almost identical set up.) Very odd to me that it's acting up now.
All good recommendations above, but another variant/option: split your render into several segments, and restart AE and/or empty cache between each segment render.
1. Reset your Ae preferences
2. Render Png sequence and set "skip existing files" in the render settings. If it fails, when you render again it will skip the already rendered frames
if that doesn't work I would hold shift as I enter the preferences. This will reveal the secret feature at the bottom of the list in preferences window. Check disable layer cache and choose 1 in purge every frame.
note: I suspect you are keying a highly compressed footage. Another advice would be to pre-render this footage to a lossless format and only then apply the effects (if it's after the fact - just replace the old for the new in the project panel by dragging new over old and hold the Alt key.)
Thanks! Worked great and was able to finish the render.