I think you're probably seeing some transparency bleed in from the edge, but you've not been terribly specific.  Do you see any hint of checkerboarding if you make all layers below invisible?

A screenshot would be helpful.

-Noel

I think it will be the bicubic resampling algorithm used when scaling that's making a 1 or 2 pixel border of grey along these edges of the mask, hence some of the original image slightly showing there.

A mask has a padding value of white or black which represents the region outside of a mask's pixels. When you downscale a mask, the padding value is brought into the region of new pixels in the mask and included in the resampling calculations. If a mask border was black and the padding is white, a gray will result.

Similarly, the transparency value of a pixel in the edge of an image is diluted by a transparency padding. Adobe do not want to fix this by making their bicubic algorithm less naive and giving the user a choice of whether a border is contaminated by padding; they say most people want bicubic resampling to give the results that Adobe's bicubic gives. They do not care that sometimes their customer does not want the contamination.

Here's an example. First there's a black layer with a new mask, over a white background. The empty mask appears white; the padding of the mask is white. Second, black is put in the mask so there's black up to the border. Third, the layer (and its mask) is downscaled. White padding has been brought into the mask and resampling has created gray pixels in the mask where black meets white. Fourth, a closeup of a gray line. Fifth is the same region of the mask. Gray in the mask allows the masked black layer to partially show against the white background as faint gray lines.

With the example that I posted, a mask with black padding would have avoided the dilution of the black of the original mask border and the resulting gray line in the image.

Black padding can be achieved by adding an empty black mask or inverting a mask which has white padding. (According to an Adobe engineer, inverting a mask shouldn't invert the padding. Should and does are different things, though. )

An empty black mask can be added by Opt/Alt-clicking the Add Mask button at bottom of Layers panel. A mask can be inverted by targeting it and pressing Cmd/Ctrl+i.

Padding contamination when scaling can be avoided by not using bicubic resampling. You can choose to use bilinear or nearest neighbour resampling, but that generally gives poorer looking results in the image.

Is there something fundamentally wrong with the concept that there's nothing outside the edges?  It seems under many conditions people just don't expect nor want "nearby values" beyond the edge of their visible canvas to influence their results.

Back when things were simple there just was nothing outside the edge of the image, and resampling didn't drag anything in.  It still works that way with the Background.

-Noel