> I'm using the animation codec

 

There's your problem. I'll ask you again to read this page.

The Animation codec is not (repeat, _NOT_) intended for playback or distribution. It is intended as an intermediate codec, to be used for movies that are passing from one piece of post-production software to another---such as from After Effects into a nonlinear editor. The Animation codec loses no image data, but it also doesn't compress very much.

For playback and distribution, you want to use a codec that does much more compression. A common example these days is H.264, often in an F4V container. The H.264 codec does lose some image data, but that is normal for a codec that shrinks files by such a large amount.

Here's some more information about the F4V format and H.264 codec.

> I'm trying to start with something as high quality as possible before Vimeo encodes it to H.264 but staying under the upload size limit.

 

If you give Vimeo a properly encoded H.264 movie, it doesn't re-compress it. Why not just do that? They have their specifications on their website.

But, if you insist on doing it the way that you're trying to, then try using the PNG video codec in a QuickTime container. At the highest quality setting, it is also lossless, but it takes up half the space as the Animation codec for a typical video. (Animation is more efficient for areas of flat color, like cartoons.)

You can also create a master using After Effects, using a lossless codec like one of those mentioned above, and then you can experiment with Adobe Media Encoder to compress to all sorts of different formats, with all sorts of different settings. There's no reason to re-render your After Effects composition just to experiment with encoding settings.