The Clone Stamp does not work that way (with a selection), so I'm thinking either you're not accurately describing what tool you're using or you're trying to use the Clone Stamp improperly (and may have crossed it up with the Patch Tool). Or maybe you're using the terminology in a misleading way (e.g., choice of clone source isn't the same as making a selection).
Can you please be a little more specific, i.e., give a step by step process describing just what you're doing? That would be helpful.
Ok, here's a specific....
A woman's lips have some small reflections from a flash. I want to remove those, replace those white spots with red from the color on the lips very nearby. I clone a sample from nearby, place the cursor over the white spot, click, and, voila, the white spot is now perfectly colored like the rest of the lips. But, sometimes that sampled red doesn't go over the white spot, even though the cursor is right there; instead, the red sample goes up onto her forehead, or into the sky, or down on her leg. It's weird. Sometimes, a whole swatch of the image is distorted, as if its sliced. Again, this does not happen all the time, just periodically.
By the way, 8GB RAM, 1GB video card, plenty of scratch space. And, this has only recently started to happen; it has never happened before over years of using Photoshop, and the clone stamp in this way....