Fellow Countrymen (and women),

I work as a graphic designer for a large outlet chain retailer which is constantly growing our base of centers.  This growth has brought a workload that used to be manageable with but two people to a never ending sprint with five.  Much of what we do is print, which is not my forte, but is also generally a disorganized, ad-hoc affair into which I am wading to try to help reduce overall strain.

Upon picking up InDesign I noted the power of the simple Data Merge function and have added it to our repetoire in mass merging data sources.  There are some critical failures I see in this as a tool going forward for our purposes, however:
1) Data Merge cannot handle information stored and categorized in a singular column well.  As an example we have centers in many cities, and each center has its own list of specific stores.  Data merge cannot handle a single column, or even multiple column list of these stores very easily and has forced us into some manual operations to concatenate the data into one cell and then, using delimiter characters, find and replace hard returns to seperate them.

2) Data Merge offers no method of alternate alignment of data, or selection by ranges.  That is to say:  I cannot tell Data merge to start at Cell1 in one column, and in another column select say... Cell 42 as the starting point.

3) Data merge only accepts data organized in a very specific, and generally inflexible pattern.

These are just a few limitations.

ON TO MY ACTUAL DILEMMA aka Convert to XML or not?

Recently my coworker has suggested we move toward using XML as a repository / delivery system that helps us quickly get data from our SQL database into a usable form in InDesign. 

I've watched some tutorials on Lynda.com and havent yet seen a clear answer to a very simple question:
"Can XML help to 'merge' large, dynamic, data sets like a list of 200 stores per center over 40 centers based off of a single template file?"

What I've seen is that I would need to manually duplicate pages, linking the correct XML entry as I go rather than the program generating a set of merged pages like that from Data Merge with very little effort on my part.  Perhaps setting up a master page would allow for easy drag and drop fields for my XML data?

I'm not an idiot, I'm simply green with this -- and it's kind of scary because I genuinely want us to proceed forward with the most flexible, reliable, trainable and sustainable solution.  A tall order, I know.  Correct me if I'm wrong, but XML is that beast, no?

Formatting the XML

Currently I'm afraid our XML feed for our centers isnt formatted correctly with the current format looking as such:

<BRANDS>

     <BRAND>

          • BrandID = xxxx

          [Brand Name]

          [Description]

          [WebMoniker]

          <CATEGORIES>

               <CATEGORY>

                    • xmlns = URL

                    • WebMoniker = category_type

          <STORES>

               <STORE>

                    • StoreID = ID#

                    • CenterID = ID#

I dont think this is currently usable because if I wanted to create a list of stores from a particular center, that information is stored as an attribute of the <Store> tag, buried deep within the data, making it impossible to 'drag-n-drop'. 

Not to mention much of the important data is held in attributes rather than text fields which are children of the tag.

Im thinking of proposing the following organizational layout:

<CENTERS>

     <CENTER>

     [Center_name]

     [Center_location]

          <CATEGORIES>

               <CATEGORY>

                    [Category_Type]

                    <BRANDS>

                         <BRAND>

                              [Brand_name]

My thought is that if I have the <CENTER> tag then I can simply drag that into a frame and it will auto populate all of the brands by Category (as organized in the XML) for that center into the frame.

Why is this important?

This is used on multiple documents in different layout styles, and since our store list is ever changes as leases end or begin, over 40 centers this becomes a big hairy monster.  We want this to be as automated as possible, but I'd settle for a significant amount of dragging and dropping as long as it is simple and straightforward.  I have a high tollerance for druding through code and creating work arounds but my co-workers do not.  This needs to be a system that is repeatable and understandable and needs to be able to function whether I'm here or not -- Mainly because I would like to step away from the responsibility of setting it up every time

I'd love to hear your raw, unadulterated thoughts on the subject of Data merge and XML usage to accomplish these sorts of tasks.  What are your best practices and how would you / do you accomplish these operations?

Regards-

Robert