After exporting the final project to h264 via Media Encoder or AE
don't use H.264 under quicktime format options in Ae. it's is buggy and low quality. use AME (which you also did)
I also checked and unchecked settings like "Use maximum render quality" and "Render at Maximum Depth" with no visible change.
these should not make a difference unless you are resizing your image in the render process and not rendering on the GPU, or require 32bit depth for further compositing (which H.264 does not support anyway - it is always 8-bit): Understanding render options in Adobe Premiere Pro - Blog - Digital Rebellion
So I tested almost every option in the h264 video settings with no success.
using match source - high bit-rate default preset should be fine.
guess is that the way the h264-compression works causes those ugly edges. Maybe it's the color contrast of red and blue? I don't know. Did anybody else notice this problem? Any ideas on how to avoid this other than using another codec? Oh and, still working with AE CC 2015.2 if that matters.
yes compression does degrade your image and contrasting pixels are a problem with mpeg compression BUT when you say that CC2015.2 is fine (you should upgrade to 2015.3 btw) this leads me to believe you are experiencing another issue which is not an encoding issue but a dynamic link bug between Ae and AME in CC2017. the bug causes AME to get the quality and resolution from the composition pop up menu in Ae. read about it here: Known issues in After Effects CC 2017
the workaround is set the resolution pop up in the composition panel to full before you add your composition to AME, or use the much better - Queue in AME feature. Try and see if that solves your problem.
Did Roei's response help you solve your issue?