Without proper system info, details about your MP settings, comp settings, render settings, effects used and so on nobody can say anything. You are losing yourself in vagueries based on hearsay and your lack of experience. It always comes down to specific details. You could simply be using a leaky effect or render to an unsuitable output format and the error not at all have anything to do with actually running out of memory.
Below are screen shots of the aforementioned settings. I receive the error as I scrub to 00;00;20 on the composition. The message displays as a large .jpg file begins to rotate out and another .jpg file rotates in. I've used a camera to pan in on this elements' transitions but am not using any effects directly. I hope this helps!
Two things come to mind:
Don't accelerate effects with Open GL: they rarely -- if ever -- work properly.
You write that this happens when a "large jpeg" rotates. Just how large is "large"? what are the horizontal & vertical dimensions of these stills?
It was originally a .jpg I saved and imported as a .psd. The image was imported at 5184x3456.
I disabled Open GL via edit>preferences>previews>enable open GL and still received the dialogue box.
The one thing I see that's amiss in your settings is the amount of memory for other applications.... just 1.5 gigs? Really? The OS by itself can eat up a gig, and open browsers are notorious memory hogs. You could be choking your OS... and if the OS ain't workin' right, ain't NOTHIN' gonna work right.
I did reduce it as an experiment to troubleshoot...typically set at 3 GB.
My composition scrubs and plays back normal up until I hit the .psd rotation and the additional element (both pictured in second screenshot). It's also a .psd at 748x118 (set as 3D).
What's interesting is that I if I toggle between 3D and non, I can bypass the error for a time. If it's left as 3D and any settings are changed, the error will display. Everything functions normally if I leave the element as 2D.
Not sure if that helps?
Does the same problem happen when you do a RAM Preview?
That ought to end the "operator error" portion of the diagnosis. If problems persist, we're going to need a lot more information: