It's hard to tell what you are trying to accomplish with this beyond simply grouping nearby elements. A bit more context of the series of events along the way would be helpful as it might be easier to approach this from a different angle.
Essentially that is all I am trying to do. If the items that are nearby could be grouped and then sent to the corresponding layer.
Here is a layout of the steps I take.
1. I select the text objects
2. I run my script to create layers based on the selected text objects content and format the layer name to a certain requirement. Callout and 3 digits.
3. I select the circle line and text that are close and send to the correct layer.
So for an example using my picture above.....
1. I select number 13
2. I run my script to create my custom layer based on my selection and get a layer name Callout 013
3. I select the number 13 and the line and circle that are right there with it
4. I send those 3 items (#13, circle, line) to the layer Callout 013
Does that help?