This is normal behavior. After Effects does not work like an editing application, and it does not play video files as an editing application would do. To get real time, you do a RAM Preview. For more information. open AE Help and refer to the topic, "Views And Previews".
Are you using RAM preview?
Not sure if this will help, but I have an issue playing back on two of my MAC workstations, even through RAM preview, if I have colour profiles enabled. When working on the MAC, I set the project to 8bit, no colour management. All the colour work is done on another PC. Once again, I have not even discussed this on the forums, I just assumed the MAC had isues of its own, and this was my work-around. If you need to get urgent playbacks for your client (and colour is not an issue yet) maybe try that?