I never tried doing this in a collection before, that may be the problem (or may not be the problem).
It should work when you reconnect photos in a folder.
I know it should. And it has in the past. But not now, for some reason. I'm sure I'm doing something wrong - but I'm not sure what!??
"Nearby" tends to mean "in the same folder". Does your collection contain images from multiple folders?