It is probably not a codec thing. That would be highly unlikely.
Without a complete description of your project and a screenshot with all properties revealed for the layers with problems we can not even guess what is going on. We need more info.
Take one of your good quality renders and use the Media Encoder to create the h.264. The Adobe Media Encoder does a much better job at encoding H.264 anyway.
Well, I'm attaching a screen shot but the reason I didn't was because it was an extremely simple project.
I can't find the same project but here's a similar one - literally import image sequence, drop it in a comp and retime.
Also I rendered using AME and not through AE.
The settings would be the same as this.
Still, if the DNxHD file looks good, use it to create your H.264 and it should also look good.
I just had another thought, are you interlacing this project? If so turn it off.
Project is not interlaced, thanks.