For anyone who's curious, this didn't really solve the problem, but it was a satisfactory work-around:
I went ahead and exported the corner-pinned, UNmasked screen replacement on its own, sitting on a transparent alpha channel. I then brought that new video layer into my composition and masked it. Since the new layer wasn't being corner pinned, the mask halted perfectly right on the visible masking line.
The problem occurs because of After Effects' fixed render pipeline: masking always happens before filters, but you required it in reverse order.
Your solution is perfectly reasonable. Another would be to precompose the layer with the corner pin applied, then work with the mask.
Another solution would be to mask on a new solid layer and use it as a track matte.