First off your description doesn't make a lot of sense. Which tool did you use to create the rectangle? Which tool are you using to modify the rectangle? Hold the Selection arrow over the tool to get the name.
Thank you for the reply.
I used the rectangle tool to draw the rectangle, and then I used the selection tool to select it. After that, the horizontal options menu near the top of the window changed to say "Path" and it contained some stroke options. One of the options was the stroke weight, and just to the right of that, what appears to be a drop down menu of brushes. In the default configuration, the last options look to be borders. I chose the one called "Thick line 1.2" and then noticed the issue I described above.
There are a couple of ways (or maybe more) to do this. Both involve seperating the object into two parts, one with a fill and no stroke and the other with a stroke and no fill. Make sure the stroked object is on top and apply the brush stroke and then lock it. Add and delete Anchor points at the four corners to get the shape shown below (magenta rectangle) which will be masked by the brushed line. You can also do this using Object>Expand Appearance to generate the two shapes from the original.
Wonderful. Thank you.
larry has given you the way to create your own border and then you can turn it into a brush if you read about brushes.
BTW the brush you wanted was under Open Libraries>Borders>Lines
Here is a relatively easy way to make a custom border and btw this should not be necessary as almost every other vector art program has such a feature.
BTW if you do not like the look of the inverted radius being abrupt at the joins you can align the stroke to the inside, just remember to add height and width to your dimension of one half the stroke
you get this type of corner
Also I am not certain why Larry thinks you need a separate fill shape and stroke shape, this is down with one shape that can be stroked and filled.
Try it Wade. If you use the brush that the OP used on a filled rectangle, the corners will extend beyond the brush.
Then the brush is useless much better off using the technique I demonstrated.
could probably be scripted with corner controls.
That answer the question, thanks Larry.
The rounded rectangle tool certainly needs a negative setting.