What effects do you have? What animations do you have? Check your composition settings to make sure it's using the standard renderer and not the ray-traced renderer.
Also, what are your actual computer specs? You've got listings for at least two system specs with different clock speeds and cores on the processor and different RAM with "configurable" options listed and no mention of what your system actually has.
Also, other than the GPU's you've listed, that is not a monster machine at all. Only 12 GB of RAM? That's not a lot for AE.
i'm using Primate Keyer and Magic Bullets Looks along with basic transitions. Ray-traced is not selected.
Mac Pro (late 2013)
16 GB of ram
250 GB HD (140 free)
1 TB external thunderbolt drive
Processor: 3.5 GHz 6-Core Intel Xeon E5
Graphics: AMD FirePro D500 3072 MB
Okay. 16 GB of RAM is a bit better No monster, but reasonable specs for AE.
What are your composition settings? (Screenshot would probably be easier than typing it all.)
What are your memory and multiprocessing settings? (Again, screenshot would be easier.)
Could we see a screenshot of your render settings and one of your output module settings?
You said something about h.264. AE is not good at encoding in h.264. Try encoding into something a lot less compressed. You can then drop that resulting file into the Adobe Media Encoder (which is good at making h.264 files) for your final deliverable. A good intermediate codec would be QuickTime with the Photo-JPEG codec at a reasonably high quality.
Also, you haven't mentioned yet, what version of AE are you using? And make your answer down to the last decimal, please. Saying, "CC" isn't helpful; saying, "Version 126.96.36.199" is much more helpful.
What version of Primatte are you using?
What version of Looks are you using?
And what about your render settings?
Looks can take a while to render depending on what settings you've chosen.
You also might consider updating Looks. I mean, I wouldn't necessarily recommend upgrading to version 3 in the middle of a project, but the 2.5 (free update) included faster rendering as one of the bug-fixes. (And the 2.5.3 update fixed some hanging in Premiere). Go here: Red Giant | Looks Downloads