Try transcoding that video with the Adobe Media Encoder into a proper production codec. (Something like QuickTime with the PNG codec.) Use the resulting file instead. See if it behaves.
But will it loose quality? and to make sure I understood you right. Do you mean converting from MP4 to MOV?
But will it loose quality?
No. That's the point of a production codec. You're taking a compressed, lossy h.264 and converting it into a lossless file. QuickTime with the PNG codec is a technically lossless format. That means that I should be able to go ten generations in and it will still look pristine.
Now, QuickTime with the Photo-JPEG codec would work too (with smaller file sizes than PNG too) and it's visually lossless, but it does lose a little, tiny bit of quality (you probably won't notice it though unless you're doing very high end work [which, obviously you're not, because you're using h.264 to begin with]).
Antonia Tanti wrote:
Do you mean converting from MP4 to MOV?
Yes, but not just any MOV. MOV is just a container. You can have all kinds of different codecs in MOV containers. Heck, you can have an h.264 in an MOV container (bad idea to use it though).
An MP4 is (usually) an h.264 codec. The H.264 codec is a lossy codec that uses interframe compression. That means that you don't have complete information for every frame. The compression guesses and interpolates data. This makes file sizes and data rates much smaller, but it also introduces higher processing requirements, compression artifacts, and (sometimes) issues when you attempt to use it in software that works on full-frame data (such as After Effects, Nuke, etc.)
Compositing software like AE, works on full information for every pixel in every frame. These interframe compression schemes require your computer and After Effects to do a heck of a lot of work behind the scenes to be able to show you video from them. Current versions of AE (really from CS5 onwards) are a lot better at dealing with this type of compression than older versions though. CS4 and earlier really struggled!
If this is really interesting to you and/or useful for the work you're doing, you should research the difference between a deliverable codec and a production codec, interframe vs. intraframe compression, etc.
If you're concerned about quality, you shouldn't be using H.264 footage at all until your create your final deliverable. A lot of cameras that internally record to that codec, offer a way to pass an uncompressed stream out to a recording deck that captures in a lossless codec.