You are likely hitting a bug in Apple's H.264 QuickTime codec. It fails badly (crashes) when there are a large number or threads being used.
I recommend not using H.264 in a QuickTime .mov container but instead creating .mp4 H.264 files through Adobe Media Encoder.
I know the setting is in there to render h264 out of AE, but it is very inefficient. After Effects renders a frame at a time and adds it to the end of a movie. h264 encoding looks at blocks of video an determines the areas that can be discarded. This helps the final product maintain a smaller file size. Rendering h.264 straight out of AE won't give you the smallest file size or the best results.
My recommendation is to either render out a high resolution movie and then encode it to h.264 using Adobe Media Encoder, Episode, or ffmpeg. If you are just looking to save a step, you might try rendering the the video using the h.264 settings in Adobe Media Encoder.
Thnx Todd and Trent. I will give that a try. I tried it on a few other machines, and they worked fine (strangely). So I am going to trash my prefs.
I know CC is moving towards AME as the render hub, but I like the render queue as record of my workflow if I open the project a year from now.