Ran into this problem with a friends video footage.
As you are on a mac you need to decompress these files first and then bring them into AE / Premiere.
Excellent article here on what to do and how to do it.
You may need to purchase a program like Clipwrap or you can use Handbrake for free.
The article above explains how to do this for Handbrake.
I know you say you can't convert as it will take too long, but I tried all sorts of work arounds and I couldn't get the audio to work until I converted the files.
Sorry I know this is a pain and not what you wanted to hear, but I tore my hair out trying to get the audio to work and converting the files was the only thing that worked for me. I genuinely hope that someone else has a better solution than mine and that it's an easy fix.
Hope this helps.
Thanks for the reply. If no one finds an easier way, i'll have to do it the time consuming way.
Just found that if you OPEN on Quicktime Player, then EXPORT it to .MOV it works. The conversion is quick anyway