Hm, I hadn't noticed that before, but I can confirm exactly what you're reporting. Worse yet, the layer mask doesn't actually take on the rounded value (it's left at whatever fractional value was in the field before hitting Return).
I can't imagine that's a feature. Edit controls just don't work that way in general.
I think someone at Adobe must have missed removing some rounding logic in the edit control when they made the function capable of handling fractional values.