5 Replies Latest reply on Jul 1, 2015 7:30 AM by Harbs.

    Combining or Merging Indesign Files (IDML/INDT)

    diggeralmu Level 1

      Greetings,

       

      I would like to inquire about a possible performance problem regarding combining multiple IDML / INDT files or copying pages via Indesign Server Scripting API.

      Note that we are using Indesign Server CS6 (with latest 8.0.2 update).

       

      Currently, we are using the following line to merge multiple indesign documents (IDML/INDT)

      document.pages.everyItem().duplicate(LocationOptions.AFTER, baseDocument.pages[lastPageIndex])

       

      Above line performs what is needed (textframes are threaded, etc), however there is a possible issue with its performance which degrades horribly especially after several merging operations and if there are several pages in the base document or the document to be copied (though the performance is more affected when the document to be copied has many pages)

       

      Testing done:

      #1 For the below test, Notice that the merging is not constant and degrades per minute (performance may possible degrade when using a larger document)

       

      start

      end

      time elapsed for pages.everyItem.duplicate

       

      operation done

      with a base of 42 page INDT document

      12:13:33

      12:15:28

      0:01:55

       

      add 42 page INDT file

      12:15:29

      12:18:35

      0:03:06

       

      add another 42 page INDT file

      12:18:37

      12:22:53

      0:04:16

       

      add another 42 page INDT file

      12:22:54

      12:28:15

      0:05:21

       

      add another 42 page INDT file

      12:28:17

      12:34:30

      0:06:13

       

      add another 42 page INDT file

       

       

      TOTAL

       

       

       

       

      0:20:51

       

       

       

      with a base of 42 page IDML document

      12:13:58

      12:16:44

      0:02:46

       

      add 42 page IDML file

      12:16:56

      12:20:59

      0:04:03

       

      add another 42 page IDML file

      12:21:11

      12:26:21

      0:05:10

       

      add another 42 page IDML file

      12:26:35

      12:32:39

      0:06:04

       

      add another 42 page IDML file

      12:32:50

      12:40:06

      0:07:16

       

      add another 42 page IDML file

       

       

      TOTAL

       

       

       

       

      0:25:19

       

       

       

      Time elapsed is 20 minutes for INDT and 25 minutes for IDML. it adds ~1 minute for each document added (for a 42 page, this duration may increase when no of pages increase).

       

      #2 For below test, we noticed that INDT seems to perform faster than IDML (though it is larger by 10x and is a binary file)

       

       

      with a base of 838 page IDML document

      14:30:44

      14:55:40

      0:24:56

       

      add 42 page IDML file

       

       

       

       

       

       

       

       

       

       

      with a base of 838 page INDT document

      15:14:55

      15:23:40

      0:08:45

       

      add 42 page INDT file

       

      #3 For below test, notice that the merging degrades by several minutes even if you just increase the number of pages.

      Notice that we added the documents according to increasing no of pages as we found out that it performs significantly faster than not (saved several hours)

      I also incorporated the doc.save and gc calls but it still doesn't help reduce the whole merging process to several minutes. (still takes 11 hours just to merge 33 documents with total of 1046 pages)

      Note that we also checked binary merging (merging small files first but the resulting time is much larger than above)

       

      with a base of 1 page IDML document

      15:58:45

      15:58:52

      0:00:07

       

      add 1 page IDML file

      15:58:53

      15:59:00

      0:00:07

       

      add 1 page IDML file

      15:59:01

      15:59:08

      0:00:07

       

      add 2 page IDML file

      15:59:09

      15:59:18

      0:00:09

       

      add 2 page IDML file

      15:59:19

      15:59:29

      0:00:10

       

      add 3 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      15:59:35

      15:59:42

      0:00:07

       

      add 4 page IDML file

      15:59:44

      15:59:52

      0:00:08

       

      add 6 page IDML file

      15:59:54

      16:00:05

      0:00:11

       

      add 6 page IDML file

      16:00:08

      16:00:21

      0:00:13

       

      add 6 page IDML file

      16:00:23

      16:00:49

      0:00:26

       

      add 7 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      16:00:53

      16:01:03

      0:00:10

       

      add 7 page IDML file

      16:01:05

      16:01:19

      0:00:14

       

      add 8 page IDML file

      16:01:22

      16:02:18

      0:00:56

       

      add 10 page IDML file

      16:02:22

      16:04:12

      0:01:50

       

      add 17 page IDML file

      16:04:17

      16:06:41

      0:02:24

       

      add 19 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      16:06:50

      16:09:14

      0:02:24

       

      add 20 page IDML file

      16:09:19

      16:12:31

      0:03:12

       

      add 20 page IDML file

      16:12:37

      16:15:53

      0:03:16

       

      add 21 page IDML file

      16:15:59

      16:21:00

      0:05:01

       

      add 26 page IDML file

      16:21:07

      16:27:25

      0:06:18

       

      add 27 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      16:27:40

      16:38:28

      0:10:48

       

      add 37 page IDML file

      16:38:38

      16:50:42

      0:12:04

       

      add 39 page IDML file

      16:50:55

      17:08:09

      0:17:14

       

      add 42 page IDML file

      17:08:21

      17:28:39

      0:20:18

       

      add 44 page IDML file

      17:28:55

      17:58:32

      0:29:37

       

      add 52 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      17:58:58

      18:26:26

      0:27:28

       

      add 52 page IDML file

      18:26:42

      19:11:57

      0:45:15

       

      add 63 page IDML file

      19:12:16

      20:08:49

      0:56:33

       

      add 64 page IDML file

      20:09:09

      21:13:22

      1:04:13

       

      add 67 page IDML file

      21:13:59

      22:39:28

      1:25:29

       

      add 67 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      22:40:15

      0:35:17

      1:55:02

       

      add 84 page IDML file

      0:35:48

      3:30:53

      2:55:05

       

      add 98 page IDML file

       

       

      TOTAL

       

       

       

       

      11:26:36

       

       

       

      Notice that I have already added $.gc and doc.save per 5 documents however the effect is lost/not significant around the 4th/5th time.

       

      Please confirm if there is something that can be done to resolve above Indesign's performance for merging / copying pages from one document to another.

      or suggest alternative code or some setting that we may have missed.

        • 1. Re: Combining or Merging Indesign Files (IDML/INDT)
          Harbs. Level 6

          $.gc rarely makes a significant difference.

           

          Are you doing doc.save() or doc.save(docFile)?

           

          The second would make a (sometimes significant) performance difference because it dumps all the undo data, while the improvement by the first would be minimal.

           

          Harbs

          • 2. Re: Combining or Merging Indesign Files (IDML/INDT)
            Harbs. Level 6

            Another thing you can try (I don't know that it will make a difference):

            document.pages.everyItem().duplicate(LocationOptions.AT_END, baseDocument)

             

            Additionally, IIRC, duplicating the pages one at a time and re-linking the text threads manually can help performance.

             

            You also might want to try duplicating spreads instead of pages. (Not sure if that will work for your documents.)

            • 3. Re: Combining or Merging Indesign Files (IDML/INDT)
              diggeralmu Level 1

              Hi Harbs,

               

              @doc.save/doc.save(file), I'm already doing doc.save(file).

               

              @LocationOptions.AT_END, I remembered that we tried this earlier and there isn't any difference.


              @duplicating pages one at a time and relinking, thanks for the suggestion. I'll try this and get back later.


              @duplicating spreads, will also look into this but not sure if applicable.

               

              Thanks

              • 4. Re: Combining or Merging Indesign Files (IDML/INDT)
                Harbs. Level 6

                Another thing worth trying:

                 

                InDesign does not clear memory/swap files until a script finishes running. I’ve seen some incredibly large swap files from the scripting engine. (like 2GB+)

                 

                If that’s your problem (highly likely), one thing that can really help is to chunk your script and process the pieces in IDLE events. When InDesign goes idle it normalizes memory.

                 

                One last suggestion: I don’t know what’s in you files, but I’d guess it’s an awful lot of text. Lots of style overrides in text greatly effects performance. If you have control of the styling, the more that can go into styles, the better.

                 

                Please report back if something helps…

                 

                Harbs

                • 5. Re: Combining or Merging Indesign Files (IDML/INDT)
                  Harbs. Level 6
                  For below test, we noticed that INDT seems to perform faster than IDML (though it is larger by 10x and is a binary file)

                  The reason is because the IDML needs to be parsed, while the InDesign file is all native (it's really a database file). IDML will ALWAYS take more time to open and it'll always be smaller.