What might be causing this?
A flakey/ misconfigured graphics driver. Update it, reset the configuration.
I've already updated the graphics card driver to the latest Nvidia driver... but resetting the config in the Nvidia control panel didn't do anything.
YES!! I found the problem!!
It was my on-board soundcard causing this slowdown! Once I deactivate the "High Definition Audio-Controller" in my device manager, everything works fast even with AE being the active window.
Edit: Alright, even though the problem can be avoided through deactivating the audio hardware, it's not really a permanent fix. As soon as I plug in another audio card (in my case an external USB sound card), the slow down comes back. Even more: When another program accesses the audio hardware (like playing MP3s with WinAmp or listening to a CD with Windows Media Player), After Effects slows down, even if the Window is inactive. So this whole thing is connected to some audio stuff...