Which version of After Effects & QuickTime player you have installed?
Which Operating System?
What are your system specifications?
We've got the latest QuickTime 7.7.9 and using 13.2 of After Effects.
Windows 7 Pro.
Intel Xeon CpU E5-2650 v2 2.6GHz
NVIDIA Quadro K4000
Encoding h.264 out of After Effects has always been a bit buggy. That's why Adobe removed the native option to do it in CC 2014. Granted, you can still do it in a QuickTime file (which is what you are trying to do, I'm guessing), but that's even buggier and Apple doesn't even support that.
I would highly suggest rendering an intermediate file out of AE and using the Adobe Media Encoder to make your H.264. You can get a better-quality product in less time that way.
Details and more info here - please do read it.