It will change whichever one (stoke or fill) is in front and active. Sounds like you probably aren't clicking on the one you think you are.
I went back and figured out what was going on. As you mention, whichever is the active option (fill or stroke) gets the color applied to it. But I feel this may have be an interaction that was overlooked by Adobe? Double-clicking the color box in the Tools or Color Panel makes that the active one and then opens the color picker, but not so within the Control Panel. It only changes if I select the arrow next to the color square first.
I avoid the color picker because I find it fairly useless (I prefer to use the Color panel or Swatches and really define by number) and potentially dangerous (you can destroy a CMYK or Spot swatch if you don't know what you're doing), so I've never tried double-clicking in the control panel like that. I would say though, that this is a bug, and I'll report it as such.
Yeah, I usually define my colors by number, too. I just meant the actual "Color Picker" window that opens up, not the "Color Space" view.
Thanks for reporting it.