My guess is that you have very little RAM, and the frames cached to RAM are being dumped to make room for something else---such as the running of the application.
How much RAM do you have, and what are your Memory & Multiprocessing preference settings?
a) You eitehr have very little RAM as Todd already said.
b) You have enabled OpenGL for the comp vieweer and set your acceleration options for effects to use it, which will invalidate the RAM cache and attempt to render everything live.
c) Your project structure is such, that some pre-comp or otehr element updates whenever you scrub, which could be either an expression calculates a new result or some footage being pumped into the media cache.
d) You have something going on with audio and your audio hardware is misconfigured.
e) Similar to b), an OpenGL driven third-party effect like Magic Bullet Looks may be misbehaving.
f) Your graphics card is misconfigured, causing general redraw errors that invalidate the cache, possibly also related to OpenGL.
g) An input device driver e.g. from a tablet, some CoDec or otehr such thing is acting weird.
Check all these things and flip the relevant switches where appropriate and update any outdated drivers. Some combination is bound to resolve this.
How would I disable "...OpenGL for the comp vieweer"?
Here is what I found works really well. Under the little preview window that you can see your animation in, there is a drop down menu that will say full. This is the graphics control. Lower graphics=more ability to scrub the green bar.