The specific example you give would not be difficult. Determining which objects (the two circles) are surrounded by a rectangle would be a fairly trivial matter of the script testing for object bounds (height and width) that are located within the height and width of the rectangle.
I doubt that's what you really want, since making such a selection would be a piece of cake anyway: Simply drag out a rectangular selection marquee with either of the selection pointers.
But AI's scripting engine does not provide any mechanism that could be used as a "collision test" or "intersection test" for paths of any given irregular shape. That would indeed be useful for many scripted things. I've been arguing that for years. I've also been arguing for years that Illustrator should offer a feature in which any path could be used as either a selection marquee or as a cutting path. But alas...
So I seriously doubt you'll find a script that does what you describe in a practical sense (working with paths of any shape, not just a rectangle). And I have difficulty imagining why you'd need one that only makes rectangular selections.
Mmh, depending on how complex the paths are, one could possibly do a proximity test for points, but the condition here realyl is, that they need to be withing reasonable ranges. This would be relatively simple, if the original selection was based on an element with straight edges, or in reverse, the original selection had a point inside the elements that are to be selected and those also only contain straight edges. Figuring out the convexity of the shape in this case is trivial (the point's position is always larger than the leftmost/bottommost point, but always smaller than the rightmost/topmost), once you have sorted the point order according to the bounding box. Everything else is unfeasible. I agree with James on that one. Unless there is provision for intersection testing, this is a helluvalot vector math to rebuild the curves, sort multiple intersections, test tangentiality and whatnot and since it would run all in script code, not native, it would be slow as hog. Nothing you would enjoy using.