That is correct. The Quick Selection tool works quickly when there is a sharp contrast, but it does not always work well. The only control you have is the size of the brush.
When it does not work, use another selection method. Hope that helps!
When you select a part of the photo make sure you drag the tool in the middle of the object, not at the edges! the Quick Selection tool will find the edges itself.
If it still does not work well you can try a smaller brush for more precision.
Hope this will work!