You are going to have to provide a visual example for anyone to be able to understand what you are talking about.
Is your box on a separate layer, surrounded by transparency?
If not, it needs to be to use a Layer Style on it.
OK, I'm new to Photoshop so I appologise if my terminology or descriptions are
a little poor.
This is a rounded box. (or that's what I meant by 'rounded box'). I created it using the
Rounded Rectangle Tool and a mask.
This is the picture with the edge effect added.
I want to have the edge effect on the rounded box but I created the edge effect using the
rectangular marquee tool and if I use that the that would cut the rounded corners off.
How do I keep the rounded box but apply the edge effect?
How is this edge effect applied, specifically? Did you do something to a layer mask to create it? It's still not clear how you're accomplishing the above (a display of your Layers panel would help).
It's possible what you want to accomplish can be achieved by rasterizing the vector mask and operating on the mask layer...
To make the rounded box I did the following.
1. Open image
2. Make a rounded rectangle with the Rounded rectangle tool
3. Double click on Shape layer and reduce fill opacity to 0
4. Select Paths, right click and Make Selection
5. Edit>Copy merged
I applied the edge effect using the following method.
1. Open image
2. Make a box about 1/2 inch inside the image using the rectangular marquee tool.
3. Enter Edit in Quick Mask (press Q)
4. Select Filter>Brush Strokes>Sprayed Strokes
5. Press Q
6. Inverse selection
You could do that a lot more directly.
Draw a rounded rectangle as a path, then make a mask with it. If you go directly to making a pixel mask (opposed to a vector mask) you can then run effects directly on the mask, as I have done above.