6 Replies Latest reply on Sep 24, 2012 6:10 AM by Robert.Ralph

    [CS5.5/6] - XML / Data Merge questions & Best practice.


      Fellow Countrymen (and women),


      I work as a graphic designer for a large outlet chain retailer which is constantly growing our base of centers.  This growth has brought a workload that used to be manageable with but two people to a never ending sprint with five.  Much of what we do is print, which is not my forte, but is also generally a disorganized, ad-hoc affair into which I am wading to try to help reduce overall strain.


      Upon picking up InDesign I noted the power of the simple Data Merge function and have added it to our repetoire in mass merging data sources.  There are some critical failures I see in this as a tool going forward for our purposes, however:
      1) Data Merge cannot handle information stored and categorized in a singular column well.  As an example we have centers in many cities, and each center has its own list of specific stores.  Data merge cannot handle a single column, or even multiple column list of these stores very easily and has forced us into some manual operations to concatenate the data into one cell and then, using delimiter characters, find and replace hard returns to seperate them.

      2) Data Merge offers no method of alternate alignment of data, or selection by ranges.  That is to say:  I cannot tell Data merge to start at Cell1 in one column, and in another column select say... Cell 42 as the starting point.

      3) Data merge only accepts data organized in a very specific, and generally inflexible pattern.


      These are just a few limitations.


      ON TO MY ACTUAL DILEMMA aka Convert to XML or not?


      Recently my coworker has suggested we move toward using XML as a repository / delivery system that helps us quickly get data from our SQL database into a usable form in InDesign. 


      I've watched some tutorials on Lynda.com and havent yet seen a clear answer to a very simple question:
      "Can XML help to 'merge' large, dynamic, data sets like a list of 200 stores per center over 40 centers based off of a single template file?"


      What I've seen is that I would need to manually duplicate pages, linking the correct XML entry as I go rather than the program generating a set of merged pages like that from Data Merge with very little effort on my part.  Perhaps setting up a master page would allow for easy drag and drop fields for my XML data?


      I'm not an idiot, I'm simply green with this -- and it's kind of scary because I genuinely want us to proceed forward with the most flexible, reliable, trainable and sustainable solution.  A tall order, I know.  Correct me if I'm wrong, but XML is that beast, no?


      Formatting the XML

      Currently I'm afraid our XML feed for our centers isnt formatted correctly with the current format looking as such:




                • BrandID = xxxx

                [Brand Name]





                          • xmlns = URL

                          • WebMoniker = category_type



                          • StoreID = ID#

                          • CenterID = ID#


      I dont think this is currently usable because if I wanted to create a list of stores from a particular center, that information is stored as an attribute of the <Store> tag, buried deep within the data, making it impossible to 'drag-n-drop'. 


      Not to mention much of the important data is held in attributes rather than text fields which are children of the tag.


      Im thinking of proposing the following organizational layout:














      My thought is that if I have the <CENTER> tag then I can simply drag that into a frame and it will auto populate all of the brands by Category (as organized in the XML) for that center into the frame.


      Why is this important?

      This is used on multiple documents in different layout styles, and since our store list is ever changes as leases end or begin, over 40 centers this becomes a big hairy monster.  We want this to be as automated as possible, but I'd settle for a significant amount of dragging and dropping as long as it is simple and straightforward.  I have a high tollerance for druding through code and creating work arounds but my co-workers do not.  This needs to be a system that is repeatable and understandable and needs to be able to function whether I'm here or not -- Mainly because I would like to step away from the responsibility of setting it up every time


      I'd love to hear your raw, unadulterated thoughts on the subject of Data merge and XML usage to accomplish these sorts of tasks.  What are your best practices and how would you / do you accomplish these operations?




        • 1. Re: [CS5.5/6] - XML / Data Merge questions & Best practice.
          Peter Spier Most Valuable Participant (Moderator)

          XML is, to put it simply, very complex. XML workflows in ID are not for the faint of heart or the inexperienced, so if you don't already know a lot about XML, you have a steep learing curve.


          There are some commercial catalog plugins that are a lot more sophiticated than Data Merge. I've never used them, but I think it might be worth your time to contact the vendors and talk to a KNOWLEDGEABLE salesperson aobut the details of your workflow and see if their product can handle it. These plugins are not inexpensive, but probably worth their weight in gold to oyour organization if you acan find one that will do the job.

          • 2. Re: [CS5.5/6] - XML / Data Merge questions & Best practice.
            Robert.Ralph Level 1

            From what I've gleaned through watching Lynda tutorials on the subject is that what I'm hoping to do is indeed possible.


            Peter, I dont disagree with you that there is a steep learning curve for me as the instigator / designer of this method for our team, but in terms of my teammates and end-users that will be softened considerably.  Even so I'm used to steep learning curves and the associated frustrations -- but I cope well with new learning and am self taught in many tools and programs.


            Flow based XML structures:

            It seems as though as long as the initial page is set up correctly using imported XML, individual data records that cascade in a logical fashion can be flowed automatically into new pages.  Basically what you do is to create an XML based layout with the dynamic portion you wish to flow in a single frame, apply paragraph styles to the different tags appropriately and then after deleting unused records, reimport the XML with some specific boxes checked (depending on how you wish to proceed).


            From there simply dragging the data root into the frame will cause overset text as it imports all the XML information into the frame.  Assuming that everything is cascaded correctly using auto-flow will cause new pages to be automatically generated with the tags correctly placed in a similar fashion to datamerge -- but far more powerful and flexible. 


            The issue then again comes down to data organization in the XML file.  In order to use this method the data must be organized in the same order in which it will be displayed.  For example if I had a Lastname field, and a Firstname field in that order, I could not call the Firstname first without faulting the document using the flow method.  I could, however, still drag and drop content from each tag into the frame and it would populate correctly regardless of the order of appearance in the XML.


            Honestly either method would be fantastic for our current set of projects, however the flow method may be particularly useful in jobs that would require more than 40 spreads or simple layouts with huge amounts of data to be merged.

            • 3. Re: [CS5.5/6] - XML / Data Merge questions & Best practice.
              Peter Spier Most Valuable Participant (Moderator)

              I'm self-taught in a number of skills, too, so I hear you. I'm too old, though, to want to take on XML.


              Good luck.

              • 4. Re: [CS5.5/6] - XML / Data Merge questions & Best practice.
                Colin Flashman Adobe Community Professional

                If the OP hasn't already purchased this book, go out and get Jim Maivald's book "A Designer's Guide to Adobe InDesign and XML: Harness the Power of XML to Automate your Print and Web Workflows". IMHO it is the best resource to date which combines InDesign and XML.


                As the OP has already pointed out, Data Merge is quite limited. Put simply, it takes a one to one relationship database and plops the information into placeholders without parsing the data. It's great for business cards, direct mailers, invites... items that have one record per item. It's not intended as a query window/output.


                XML is the other solution via InDesign but as Peter has pointed out, the learning curve is STEEP, and out of all of my clients, none of them actually prepare XML for me to use for artwork – always excel spreadsheets or tab files from filemaker pro.


                My suggestion would be to have a look at the list of third party plugins http://www.adobe.com/products/indesign/indepth.displayTab3.html specifically XMPie, Em Software, or Cacidi systems. Don't expect them to be cheap either.


                The one question that I can't grasp from the OP's comments though is: what is trying to be created? catalogues, direct mail items, WHAT? different items may require different solutions. Lots of detail without knowing the brief so far, much like asking what screws to buy for a building project without knowing what is to be built (a birdhouse, car, aircraft carrier).


                What I will say is that XML hasn't become any easier over the versions and CS6 is no exception, and I'd like to think I'm an intermediate-advanced user of InDesign. Trying to make something foolproof is one thing, but I guarantee that even if something appears foolproof, the world will provide a dumber fool who won't get it.


                Opinions and sarcasm aside, Jim Maivald's book is worth the purchase and should answer most of the OP's XML questions.

                • 5. Re: [CS5.5/6] - XML / Data Merge questions & Best practice.
                  Laubender Adobe Community Professional & MVP

                  @Ralph – could also be, that XML alone (or transforming XML with XSLT beforehand) will get you just  90% to your goal. The rest must be scripted with JavaScript (ExtendScript). So I recommend finding a consultant who is savvy in XML workflows PLUS doing JavaScript with XML.



                  • 6. Re: [CS5.5/6] - XML / Data Merge questions & Best practice.
                    Robert.Ralph Level 1



                    Mostly what we work on that would benefit from dynamic merging capabilities would be Directories, coupon books, store listings in various formats and intents.  It isnt that the information on a per document basis is so tough to typeset by hand... however we have 40+ documents (and growing year per year) to create for 40+ individual outlets with different store listings.  These store listings are subject to change month per month and so a catalogue we used 2 months ago may have 10 different store changes, multiplied by different centers which increases the rate of error tremendously.  Not to mention that for certain events that we hold certain stores opt into honoring coupon offers and so they need to be added to a different (non-directory) list of participants, usually in the same document. 


                    It isnt necessarily that our issue is that we have a huge amount of data, as 200 stores x 40 centers really is smaller compared to a full blown catalogue, however it is the dynamicity that's the issue. 


                    Currently about twice every quarter a single designer spends upwards of 16hrs of time developing these documents by-hand with multiple revisions.  I want to streamline this so that the designers only real work is formatting the imported information so it looks good with a minimal in back and forth proofing for accuracy.  My hope is that what once took 16hrs takes perhaps 2hrs.




                    Luckily the guy that codes our internal data warehouse is a flexible guy. HE is the one that wants to use XML to dump out information to indesign, and he already suggested using javascripting to create plugins to assist in that regard.  He's pretty deadset on making it as flexible and hands-off as possible.  Both he and I live by the motto that "The smart man is the lazy man" -- Rather do it well once so that every time after you're not killing yourself to do it again.


                    So we have the programming know-how and ability to create a very specifically formatted output because this is all internal.