In my old darkroom days I had a punch that made this type of edge - it was known as a "Deckled Edge"
I made something similar using Corel Draw (Shh!) but never figured out a way to so this in Photoshop.
You could use a square Brush rotated and spaced accordingly.
But if it’s a one time task creating a Brush Preset my not be worth the trouble.
Does anyone know what this effect is called
Yes Pinking is another term and you can buy Pinking Shears to do this mechanically
Might be a tiny bit tedious, but drawing a mask with the Pen Tool, possibly guided by, well, Guides, shouldn't be too terribly difficult. I'm not at my Photoshop system right now to check it, but for folks familiar with editing Paths, there are probably shortcuts that would help make the path exact.