It's the center point of the rectangle; useful for alignment and movement. Can be turned off in the Attributes panel (icons on the left). The rectangle created with the Pen tool can have one as well; just needs to be turned on.
Magic! Thanks for putting that to rest for me!