Check the MPEG level in your H.264 settings. Certain levels only support contrained, fixed resolutions and framerates.
H.264 using AME is limited to 4K. if you want more you could render Quicktime animation in Ae and use https://handbrake.fr/ (free) to convert. if you do this a lot you could buy the new script AfterCodecs and do it right from Ae: AfterCodecs - aescripts + aeplugins - aescripts.com (currently at 55$)