Data merge, as I understand it, cannot do what you want with just a two row by two column file, but it might be able to do it with a 40 row file if you then print thousands of copies of each page, or you could use a two row by 20 column file for a single-record per page merge, again printing by the thousands.
But I presume this really won't work because your real data is more complex, and you need different quantities of each image. You might be able to script page duplications, if there really are page duplications, but it would be more efficient to print multiples of a single page in that case, anyway.
Question3. Is there a more efficient way for indesign to see what really is 2 files vs how it currently seems to see them which is as 60,000 files. I ended up breaking this job into 3 groups of 20,000 images and the output time to PDF ended up taking a combined 8 hours...totally unacceptable. Especially when i've seen other applications do even more in less than 10 minutes.
what are the other applications, that are doing this in less than 10 minutes?
I don't think you can speed up InDesign or InDesign Server to create 60,000 PDFs in less than 10 minutes.
Keep in mind that the InDesign engine has never been built in order to absorb so many data. With XML you could hypothetically get the result for a hundred of items, say a thousand but then InDesign would collapse.
Given the scheme you describe I would consider other tools indeed. Let me PM for details.
by scripting you could build thousands of InDesign files one after another where the document windows are not showing. Don't try to do it with one InDesign document using DataMerge. Build the documents out from one template, read your CSV file by scripting to place the images and other variable data to populate prepared graphic frames and text frames with contents and then export to PDF. Close the active document without saving. I think, that will speed up the task tremendously.
If you can, distribute the task to several licensed desktop versions of InDesign that run in parallel.
It cannot be done in ten minutes, though, but you might be able to cut some hours.
Really trying to create a single PDF with 60,000 distributed records X number across and Y number around per page. Picture printing of something like auction/carnival tickets each with a graphic background that says the same thing but each ticket having a unique number.
I'll give this a try and get back to this thread. I read somewhere else that it might also be helpful to not export the PDF immediately but rather after the merge has completed. I actually confirmed that a 3rd party software that leverages PDF VT was able to take a 60,000 line csv file with each line calling a different barcode was able to output a PDF VT file with all of those unique records in 4 minutes. Using the same amount of data with then non-Server version of Indesign and exporting a standard PDF X4 file took 8 hours. Incredible. Will keep working on this over the coming days.