all the variables involved in render settings are not something I fully grasp but I can tell you that size is not necessarily unusual.
I would guess it has something to do with either the bit rate or AE just not being all that great at compression.
I render uncompressed quicktime out of after effects and a 4 minute video could be as large as 15 or 20gigs (give or take).
I bring them into Media Encoder and render them as H264 with a 6,000 to 8,000 bit rate and they're usually in the neighborhood of 3 or 400mb.
Basically you should compress your final renders in an external program. The company I work for prefers Apple Compressor over Adobe Media Encoder but i'm on a PC so I use Adobe ME.
Rendering H.264 out of AE was buggy so they removed it.
You found the one remaining way to do it, but QuickTime H.264 has been buggy for a really long time and Apple doesn't support it.
That would probably explain your issue.
As my favorite fish says, it's best to render an intermediate file out of AE and then use the Adobe Media Encoder to make your H.264 (and make a proper one with the MP4 extension). Alternatively, you can send your AE comp directly to the Adobe Media Encoder to render. There are advantages and disadvantages to both approaches.
Thanks so much @Gutter-Fish and @Szalam!
Adobe Media Encoder worked wonders! The same files came out to 300 MB for the same resolution, H.264 with the proper .mp4 extension, and a high bit rate.