6 Replies Latest reply on Oct 16, 2016 2:43 PM by jefft34718921

    Massive DataMerge 50,000+ CSV

    jefft34718921

      The data merge that we are doing is actually slightly more complicated that i'm going to outline right now. I just want to strip my problem down into it's most basic parts before I tackle my larger problem. My question for the community is as follows:

       

      Note: We are not currently using the SERVER version of Indesign just a standard (most current) version of regular Indesign CC.

       

      I'm very well versed in DataMerge I've been doing it for quite a while. We've actually been able to programmatically build everything that we need thru scripting all the way from setting up an initial image, defining variable data placeholders, performing proper rotation, step and repeat etc. etc. no big deal. I'm also familiar with some third party systems that leverage adobe's server engine that seemingly have access to more powerful step and repeat functions, fast vs. rich files, PDF/VT, etc...

       

      That being said I believe we've hit what i'm assuming is a ceiling for our version of Indesign. We have a project that recently required an approximately 60,000 row CSV file. To keep this first part of my question simple let's just say that it's two graphics stepped and repeated on a sheet let's just say 4 across and 5 around (total of 20 images per page). The data looks like the image below on the left. Very crude the same graphic over and over again called maybe 30,000 times then halfway thru it simply switches to the second graphic which repeats itself another 30,000 times until the merge hits the last row and finishes up.

       

      Question1. Can I do something like the "Proposed CSV" either via data merge or even XML without going to some third party?

      Question2. Can I do something like the "Proposed CSV" if i upgraded to InDesign Server without going to a third party?

      Question3. Is there a more efficient way for indesign to see what really is 2 files vs how it currently seems to see them which is as 60,000 files. I ended up breaking this job into 3 groups of 20,000 images and the output time to PDF ended up taking a combined 8 hours...totally unacceptable. Especially when i've seen other applications do even more in less than 10 minutes.

       

      Asset 1.png

       

      Any help would be appreciated. And as a reward for possibly answering this I have a followup question:)

       

      regards,

       

      -JEFF

        • 1. Re: Massive DataMerge 50,000+ CSV
          Peter Spier Most Valuable Participant (Moderator)

          Data merge, as I understand it, cannot do what you want with just a two row by two column file, but it might be able to do it with a 40 row file if you then print thousands of copies of each page, or you could use a two row by 20 column file for a single-record per page merge, again printing by the thousands.

           

          But I presume this really won't work because your real data is more complex, and you need different quantities of each image. You might be able to script page duplications, if there really are page duplications, but it would be more efficient to print multiples of a single page in that case, anyway.

          • 2. Re: Massive DataMerge 50,000+ CSV
            Laubender Adobe Community Professional & MVP

            jefft34718921 wrote:


            Question3. Is there a more efficient way for indesign to see what really is 2 files vs how it currently seems to see them which is as 60,000 files. I ended up breaking this job into 3 groups of 20,000 images and the output time to PDF ended up taking a combined 8 hours...totally unacceptable. Especially when i've seen other applications do even more in less than 10 minutes.

             

            -JEFF

            Hi Jeff,

            what are the other applications, that are doing this in less than 10 minutes?

             

            I don't think you can speed up InDesign or InDesign Server to create 60,000 PDFs in less than 10 minutes.

             

            Regards,
            Uwe

            • 3. Re: Massive DataMerge 50,000+ CSV
              Loic.Aigon Adobe Community Professional

              Keep in mind that the InDesign engine has never been built in order to absorb so many data. With XML you could hypothetically get the result for a hundred of items, say a thousand but then InDesign would collapse.

              Given the scheme you describe I would consider other tools indeed. Let me PM for details.

               

              Loïc

              • 4. Re: Massive DataMerge 50,000+ CSV
                Laubender Adobe Community Professional & MVP

                Hi Jeff,

                by scripting you could build thousands of InDesign files one after another where the document windows are not showing. Don't try to do it with one InDesign document using DataMerge. Build the documents out from one template, read your CSV file by scripting to place the images and other variable data to populate prepared graphic frames and text frames with contents and then export to PDF. Close the active document without saving. I think, that will speed up the task tremendously.

                 

                If you can, distribute the task to several licensed desktop versions of InDesign that run in parallel.

                It cannot be done in ten minutes, though, but you might be able to cut some hours.

                 

                Regards,
                Uwe

                • 5. Re: Massive DataMerge 50,000+ CSV
                  jefft34718921 Level 1

                  Really trying to create a single PDF with 60,000 distributed records X number across and Y number around per page. Picture printing of something like auction/carnival tickets each with a graphic background that says the same thing but each ticket having a unique number.

                  • 6. Re: Massive DataMerge 50,000+ CSV
                    jefft34718921 Level 1

                    I'll give this a try and get back to this thread. I read somewhere else that it might also be helpful to not export the PDF immediately but rather after the merge has completed. I actually confirmed that a 3rd party software that leverages PDF VT was able to take a 60,000 line csv file with each line calling a different barcode was able to output a PDF VT file with all of those unique records in 4 minutes. Using the same amount of data with then non-Server version of Indesign and exporting a standard PDF X4 file took 8 hours. Incredible. Will keep working on this over the coming days.