Good Alternative to Data Merge?

[RaizelDesign]

In my work I routinely have to create data merges of documents of 4+ pages with upwards of 200 records and multiple data fields. I know how to do this (convert the excel spreadsheet to .csv) and how to get around most bugs. I don't usually run into issues with the merge. However, it takes about an hour - on a good day - for the saved merged indesign file to export to pdf. I'm sure if I had a more powerful computer it would be a bit faster, but that's not an option at the moment.

Does anyone know of a good alternative to data merge for large files that doesn't take quite as long and is less prone to crashing?

Thanks!

[MW Design]

Sounds like you are merging direct to PDF? Try just merge to a new document, PDF that new document. You don't need to save the new document unless you want to.

I hardly ever use ID's native merge, except for helping around here. I do merge a lot using Em Software's data merge plug-in, though. I wouldn't be without it (as well as the XTags plug-in). Far more extensible and it doesn't build the merge in a single frame per record.

Mike

[RaizelDesign]

Hey, thanks, but no, I'm not merging directly to pdf. Can I have some more info on that plug-in, though?

[MW Design]

If you just create a non merged document with as many pages and place the design elements on these pages, does it still take so long? I would suspect so as there isn't anything special about the merge text frames themselves.

What are the design elements? Types of files, native ID elements, or ?

Mike

[RaizelDesign]

So, the last time this happened, I was creating a program guide for a conference for about 200 people. The people attending were going to be at different events at certain times, staying in different places, etc. And there's a personalized letter to each attendee.

I laid out one guide (a bifold - so essentially four pages) with a full-color (raster) background, a (raster) logo on the front, (vector) map on the back, and a couple more vector elements. Nothing is pasted in, all files were linked properly, paragraph and character styles were applied with no overrides.

All the changing elements were text changes on three of the four pages, although one of the text boxes did have a color fill.

You're probably right that it's slow more because it ends up being an 800 page document with semi-complex imagery on each page. So I guess my question should really be: does anybody have a way to speed up pdf exports or do you think taking over an hour for the pdf to export is perfectly normal and there's nothing that I can do to change it?

Thanks!

[MW Design]

800 pages that are design heavy?

The largest ID file I have done was a 400+ page catalog. Lots of images, of course. I don't recall it taking near as long. Same with a very graphic-intensive book that was in the couple hundred page range recently finished. Its pdf was very large due to the images and it took about 10 minutes.

But that is all apples and orqnges. It would really take someone using your merged file to see what it takes them for a comparison.

I don't know if splitting the merge up into say half or thirds would be any faster in total.

So I don't know what to say. Sorry.

Mike

[John Mensinger]

RaizelDesign  wrote

You're probably right that it's slow more because it ends up being an 800 page document with semi-complex imagery on each page. So I guess my question should really be: does anybody have a way to speed up pdf exports or do you think taking over an hour for the pdf to export is perfectly normal and there's nothing that I can do to change it?

Imagine the read/write load upon export for whatever number of linked elements you've got in those 4 pages, multiplied by the 200 copies generated by the merge.

For a job like that, I'd consider economizing that activity by building the merge-destination document without all the linked images and graphical treatments, then install all that in the merged document by putting just one copy of it on the master page(s).

[Stephen_A_Marsh]

Ensure that only the variable elements are being merged. Don’t increase the overhead by forcing the merge to also deal with static elements.

All static elements can be removed from the merge pages and be placed on temporary dedicated master page/s that are not currently applied to the file.

Once the merge has completed, apply the static content master pages to the merge master or final merged pages.

Prepression: InDesign – Fast Data Merge

[RaizelDesign]

Thanks. I think I'll try this next time. (Essentially you're saying to delete all background elements, imagery, and non-changing text, leaving them on the master page and then apply the master only after completing the merge, correct?)

[RaizelDesign]

Will this help with the pdf export, though? Because I don't usually have issues with the actual merge.

[Stephen_A_Marsh]

Apologies, I misread your post and thought that the bottleneck was in the merge, not the export.

However it is possible to combine PDF files in a similar fashion, which could be an option if you had a “static” PDF and the 200+ variable PDF files. This could happen at print output if the DFE/RIP supports variable data processing, or perhaps upstream in the PDF creation process (I would not recommend doing this manually though). It would likey depend on the stacking/layering in how this would pan out.

Try restarting your computer first and close down as many apps and non essential background processes as possible. If on MS Windows and you are not using a SSD, then perhaps check your disk fragmentation.

I would also try MW Design’s suggestion to try breaking up the PDF export into smaller batches. Open up the old job start the export of the entire job and note the start time, then go to lunch or leave for the day. The saved time on the final PDF will tell you how long the unattended process took. Then compare with a smaller subset, say 1 /4 and compare and then multiply the result by 4 and compare to the original full export.

[RaizelDesign]

I would also try MW Design’s suggestion to try breaking up the PDF export into smaller batches. Open up the old job start the export of the entire job and note the start time, then go to lunch or leave for the day. The saved time on the final PDF will tell you how long the unattended process took. Then compare with a smaller subset, say 1 /4 and compare and then multiply the result by 4 and compare to the original full export.

So I actually did it in two batches, and that was about an hour for both running at the same time - which in hindsight may not have been the smartest thing I've done. The thing is, I can't actually start an export and then just go to lunch. My boss doesn't take too kindly to wasted time so I tend to work on other projects in photoshop or illustrator while the export is running in indesign - again, probably not the smartest way of working, but, hey, at least this way I'm still employed .

[Dov Isaacs · Accepted solution]

The problem is totally due to bottlenecks in InDesign's PDF export. No amount of additional RAM or higher speed disk will make all that much difference.

One solution, albeit pricey, is to check out the products by XMPie (a division of Xerox) which provides a major plug-in for InDesign with functionality similar but much, much more advanced that InDesign's native data merge and lightening speed PDF export. Actually, the PDF export is optimized around production of PDF/VT which itself is layered over PDF/X-4. The resultant files are exceptionally small since common graphic assets appear in the final PDF file only once. Depending upon the complexity of your template, you could easily get the merged PDF at rates in excess of a hundred records / minute for a 8 to 12 page variable data booklet.

And yes, I personally tried this and measured the results.

          - Dov

[RaizelDesign]

One solution, albeit pricey, is to check out the products by XMPie (a division of Xerox) which provides a major plug-in for InDesign with functionality similar but much, much more advanced that InDesign's native data merge and lightening speed PDF export.

Thanks, I've heard of this. Unfortunately, it's not an option at the moment, but it's good to know for the future that it can do a merge that fast.

[soundchaser]

There are  Acrobat plugins such as Fusion Pro that enable merging data and/or images in a pdf file and exporting to various file formats (PDF-VT, PPML, VIPP and others) for final output, both print and web. This method takes a few minutes or less to create the initial export for thousands of records. More importantly, when outputting the final product the file formats listed are designed to RIP static elements in the exported file once (versus 800 times for an 800 page merged file) and only the variable elements remain to process, significantly reducing RIP times which is where the real bottleneck will appear.