You're probably getting that error because of a frame size/codec conflict.
Your comp is 900x600, and the MainConcept codec is looking for a frame size of 720x480 (NTSC DV).
Using the Quicktime wrapper to export the H.264 is a fine optioin, however, AE will not do as good of a job compressing to that codec as Quicktime Pro, or a dedicated compression application would do. AE can only cache a limited number of frames for the compression process (frames before and after the current frame), and as a result, the compression quality and filesize will not be as small as they could be by using a different tool for the job.
As such, your supposition about rendering out an uncompressed/losslessly compressed movie and then running it through QT Pro (or another compression application) yielding superior results was a correct one.
As Andrew said, the Mainconcept/ AME stuff works with fixed sizes and matrices. Therefore exporting a QT movie using anotehr CoDec and generating the H.264 output from QT Pro seems the easiest way of achieving peace. If you're feeling adventurous, you could render at a standardized HD res and use tools like SUPER© to transcode and resize your stuff, but that realyl depends. For short segments it's usually not worth the compromises in quality. It only has value for long stuff, where re-rendering in QT Pro would cost a lot of time...