I am not sure I understand your logic- "it selects edges like the brush with the auto mask turned off."
So follow with some discussion to see if you do really have a problem-
With the Automask 'off' the brush does not select edges!
1. With Automask 'off', the brush changes ALL pixels under the brush stroke.
So Brush on the blue sky area with automask 'off'- the green overlay shows where the large brush adjusts.
2. With Automask 'on', the brush centre cross hair chooses the image pixel tone to change, and only similar tones are changed by the brush stroke.
So Brush on the blue sky area with automask 'on'- and the green overlay shows where only the blue sky is adjusted- as that is the tone where I first clicked the brush to choose the pixel tone. (I did not click the brush on the red area!)
If the color, colors, of the edges are similar to what you have first selected yes there will be a wash over into that area. In the example above the colors are completely different so LR can pick them out. This also happens in Photoshop or any other image editing software with this or similar feature.
For example...i was trying to select the darker hair (centre cross hair on the hair, not bleeding into collar or highlights) area of this image to change his hair tones without trying to trace free hand with the auto mask turned off, but as you can see..the auto mask is turned on and not recognising the pixel difference through out the file. Its feathering from the hair through to the collar of the jacket which is something i don't want, especially when selecting white backgrounds for commercial product shoots.
Thanks for posting help, i hope this can clarify my issue.
Yep, looks like your brush has a 'bug'. It does not look to be working as it should. Automask should not select the collar in your image (if you never click the mouse cursor on the collar or a light pixel) and you obviously know how it should work.
By the way-Thanks for screen-clips.
As an average ability user I do not have an answer, other than the often stated 'Adobe standard' answers- Re-install, delete preferences file, etc.
Keep this post going if you don't have an answer.
I just did an exercise and experienced a similar problem-
The brush worked better with automask if I magnified the image larger and kept the brush small. Also try setting density to max 100.
So we know LR is not the perfect tool for "Selections" as is Photoshop.
I'm seeing this also. even with very different colors.
I have yet to understand fully how Automask works.
In a Lightroom Help pdf it states that it selects "color"
In my Scott Kelby book he states "it detects edges"
So maybe Automask looks at both color and luminosity when determining which pixels to adjust.
It doesn't explain the problem I think we are all (three of us, anyway) seeing. And has it changed with LR updates?