13 Replies Latest reply on Jul 9, 2015 2:16 PM by absqua

    Combining or Merging Indesign Files (IDML/INDT)

    diggeralmu Level 1

      Greetings,

       

      I would like to inquire about a possible performance problem regarding combining multiple IDML / INDT files or copying pages via Indesign Server Scripting API.

      Note that we are using Indesign Server CS6 (with latest 8.0.2 update).

       

      Currently, we are using the following line to merge multiple indesign documents (IDML/INDT)

      document.pages.everyItem().duplicate(LocationOptions.AFTER, baseDocument.pages[lastPageIndex])

       

      Above line performs what is needed (textframes are threaded, etc), however there is a possible issue with its performance which degrades horribly especially after several merging operations and if there are several pages in the base document or the document to be copied (though the performance is more affected when the document to be copied has many pages)

       

      Testing done:

      #1 For the below test, Notice that the merging is not constant and degrades per minute (performance may possible degrade when using a larger document)

       

      start

      end

      time elapsed for pages.everyItem.duplicate

       

      operation done

      with a base of 42 page INDT document

      12:13:33

      12:15:28

      0:01:55

       

      add 42 page INDT file

      12:15:29

      12:18:35

      0:03:06

       

      add another 42 page INDT file

      12:18:37

      12:22:53

      0:04:16

       

      add another 42 page INDT file

      12:22:54

      12:28:15

      0:05:21

       

      add another 42 page INDT file

      12:28:17

      12:34:30

      0:06:13

       

      add another 42 page INDT file

       

       

      TOTAL

       

       

       

       

      0:20:51

       

       

       

      with a base of 42 page IDML document

      12:13:58

      12:16:44

      0:02:46

       

      add 42 page IDML file

      12:16:56

      12:20:59

      0:04:03

       

      add another 42 page IDML file

      12:21:11

      12:26:21

      0:05:10

       

      add another 42 page IDML file

      12:26:35

      12:32:39

      0:06:04

       

      add another 42 page IDML file

      12:32:50

      12:40:06

      0:07:16

       

      add another 42 page IDML file

       

       

      TOTAL

       

       

       

       

      0:25:19

       

       

       

      Time elapsed is 20 minutes for INDT and 25 minutes for IDML. it adds ~1 minute for each document added (for a 42 page, this duration may increase when no of pages increase).

       

      #2 For below test, we noticed that INDT seems to perform faster than IDML (though it is larger by 10x and is a binary file)

       

       

      with a base of 838 page IDML document

      14:30:44

      14:55:40

      0:24:56

       

      add 42 page IDML file

       

       

       

       

       

       

       

       

       

       

      with a base of 838 page INDT document

      15:14:55

      15:23:40

      0:08:45

       

      add 42 page INDT file

       

      #3 For below test, notice that the merging degrades by several minutes even if you just increase the number of pages.

      Notice that we added the documents according to increasing no of pages as we found out that it performs significantly faster than not (saved several hours)

      I also incorporated the doc.save and gc calls but it still doesn't help reduce the whole merging process to several minutes. (still takes 11 hours just to merge 33 documents with total of 1046 pages)

      Note that we also checked binary merging (merging small files first but the resulting time is much larger than above)

       

      with a base of 1 page IDML document

      15:58:45

      15:58:52

      0:00:07

       

      add 1 page IDML file

      15:58:53

      15:59:00

      0:00:07

       

      add 1 page IDML file

      15:59:01

      15:59:08

      0:00:07

       

      add 2 page IDML file

      15:59:09

      15:59:18

      0:00:09

       

      add 2 page IDML file

      15:59:19

      15:59:29

      0:00:10

       

      add 3 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      15:59:35

      15:59:42

      0:00:07

       

      add 4 page IDML file

      15:59:44

      15:59:52

      0:00:08

       

      add 6 page IDML file

      15:59:54

      16:00:05

      0:00:11

       

      add 6 page IDML file

      16:00:08

      16:00:21

      0:00:13

       

      add 6 page IDML file

      16:00:23

      16:00:49

      0:00:26

       

      add 7 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      16:00:53

      16:01:03

      0:00:10

       

      add 7 page IDML file

      16:01:05

      16:01:19

      0:00:14

       

      add 8 page IDML file

      16:01:22

      16:02:18

      0:00:56

       

      add 10 page IDML file

      16:02:22

      16:04:12

      0:01:50

       

      add 17 page IDML file

      16:04:17

      16:06:41

      0:02:24

       

      add 19 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      16:06:50

      16:09:14

      0:02:24

       

      add 20 page IDML file

      16:09:19

      16:12:31

      0:03:12

       

      add 20 page IDML file

      16:12:37

      16:15:53

      0:03:16

       

      add 21 page IDML file

      16:15:59

      16:21:00

      0:05:01

       

      add 26 page IDML file

      16:21:07

      16:27:25

      0:06:18

       

      add 27 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      16:27:40

      16:38:28

      0:10:48

       

      add 37 page IDML file

      16:38:38

      16:50:42

      0:12:04

       

      add 39 page IDML file

      16:50:55

      17:08:09

      0:17:14

       

      add 42 page IDML file

      17:08:21

      17:28:39

      0:20:18

       

      add 44 page IDML file

      17:28:55

      17:58:32

      0:29:37

       

      add 52 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      17:58:58

      18:26:26

      0:27:28

       

      add 52 page IDML file

      18:26:42

      19:11:57

      0:45:15

       

      add 63 page IDML file

      19:12:16

      20:08:49

      0:56:33

       

      add 64 page IDML file

      20:09:09

      21:13:22

      1:04:13

       

      add 67 page IDML file

      21:13:59

      22:39:28

      1:25:29

       

      add 67 page IDML file

       

       

       

       

      perform $.gc  and doc.save into temp IDML

      22:40:15

      0:35:17

      1:55:02

       

      add 84 page IDML file

      0:35:48

      3:30:53

      2:55:05

       

      add 98 page IDML file

       

       

      TOTAL

       

       

       

       

      11:26:36

       

       

       

      Notice that I have already added $.gc and doc.save per 5 documents however the effect is lost/not significant around the 4th/5th time.

       

      Please confirm if there is something that can be done to resolve above Indesign's performance for merging / copying pages from one document to another.

      or suggest alternative code or some setting that we may have missed.

       

      Thanks

        • 1. Re: Combining or Merging Indesign Files (IDML/INDT)
          Dirk Becker Level 4

          I'd watch memory statistics - both from outside / OS, and inside. For example I recently tracked down a memory leak in an own plugin using OSX instruments.app. Then the InDesign Server scripting object model also exposes some statistics. A debug build as available to plugin developers might also give more insights.

          For more details I'd have to dig deeper / beyond the scope of a forum post, as I haven't worked with InDesign Server since 2012.

          Btw, $.gc() only has effects within the javascript engine.

           

          In the meantime, could you repeat the test with a server restart rather than $.gc()? Not really appealing, but a server is invoked from the outside anyway...

          You probably already ruled out the use of the books feature to combine documents into a single output.

           

          Dirk

          • 2. Re: Combining or Merging Indesign Files (IDML/INDT)
            Vamitul Level 4

            If using document.pages.everyItem().duplicate(LocationOptions.AFTER, baseDocument.pages[lastPageIndex])
            try doing a document.save(document.fullName) every 50 or so operations, as that will clear the undo stack and significantly improve performance.

            • 3. Re: Combining or Merging Indesign Files (IDML/INDT)
              Dirk Becker Level 4

              Hmm, InDesign Server and undo stack?

              Besides save() has already been mentioned/tried by the OP.

              • 4. Re: Combining or Merging Indesign Files (IDML/INDT)
                diggeralmu Level 1

                I'm doing document.save already for every 5 documents (please refer to last table in my post). It reduces the times spent for the new iteration but doesn't have an effect on the 4th and 5th iteration.

                • 5. Re: Combining or Merging Indesign Files (IDML/INDT)
                  diggeralmu Level 1

                  Hi Dirk,

                  Dirk Becker wrote:

                   

                  I'd watch memory statistics - both from outside / OS, and inside. For example I recently tracked down a memory leak in an own plugin using OSX instruments.app. Then the InDesign Server scripting object model also exposes some statistics. A debug build as available to plugin developers might also give more insights. For more details I'd have to dig deeper / beyond the scope of a forum post, as I haven't worked with InDesign Server since 2012.

                  Ah sorry, forgot to mention that we are interfacing with the Indesign Server using SOAP (using Java) where javascript file/commands are passed. (similar to executing javascript commands in extendscript except that the commands are passed to indesign server via its SOAP webservices)

                   

                  Dirk Becker wrote:

                   

                  Btw, $.gc() only has effects within the javascript engine.

                  Can you advise if $.gc does/doesn't work given our setup (SOAP but still using javascript)? I'm not sure  if "javascript engine" still applies to this setup or not.

                   

                  Dirk Becker wrote:

                   

                  In the meantime, could you repeat the test with a server restart rather than $.gc()? Not really appealing, but a server is invoked from the outside anyway...

                  You probably already ruled out the use of the books feature to combine documents into a single output.

                   

                  Dirk

                  The batch program is executed from a different machine so I don't think this is possible to terminate/restart the process in another server? (as stated above, we are sending javascript commands via Indesign servers SOAP webservices)

                   

                  Thanks

                   

                  • 6. Re: Combining or Merging Indesign Files (IDML/INDT)
                    Loic.Aigon Adobe Community Professional

                    Hi all,

                     

                    Just a 2 cts thought, would that make sense to use idml syntax to combine the documents outside of IDS? I know far easier to say it than do it but in terms of performance, I guess that not using InDesign engine to nest the files could offer dramatic improvements.

                     

                    Loic

                    • 7. Re: Combining or Merging Indesign Files (IDML/INDT)
                      Dirk Becker Level 4

                      Merging IDML might be not trivial, for example if the same styles have different IDs, or different settings.

                      • 8. Re: Combining or Merging Indesign Files (IDML/INDT)
                        Dirk Becker Level 4

                        How do you start the server? If you go thru OSX launchd - I think the server package has an example for that, the next SOAP request should start it automatically after a quit or crash. Actually you can invoke quit() as separate request.


                        SOAP is just a transport media for your ExtendScript requests, so for InDesign you actually are using ExtendScript. $.gc()  is about objects within the ExtendScript engine, but there won't be too many when you work at the described level of pages copied across documents.

                         

                        Most of that work will be performed by the C++ internals of InDesign. That does not use a deferred garbage collection, it's just reference counters on the objects. And if there really is a leak - as I suspect - you can only tell from memory hitting the limits, internal statistics etc. Not easy as there also are caches that are only reclaimed on demand.


                        • 9. Re: Combining or Merging Indesign Files (IDML/INDT)
                          diggeralmu Level 1

                          Hi,

                           

                          We already tried to checked that lead of not using Indesign to combine IDMLs but we stopped checking due to the reason as stated earlier by Dirk, merging outside of Indesign using XML / XSLT is not trivial as multiple indesign files may have different ids for each resource (where there are many type of resources e.g pages, fonts, strokes, colors, stories, spreads, tags, etc). It is not straightforward to merge these different type of resources and keep track. It would require several days (to months if not full time) to ensure that there are no problems in the XML merging tool.

                           

                          Thanks

                          • 10. Re: Combining or Merging Indesign Files (IDML/INDT)
                            diggeralmu Level 1

                            We are starting the Indesign server via the command line or as a windows service (server is Windows Server 2014). Not familiar with OSX features but I think there is no such functionality in windows where a HTTP SOAP request can start a service (as the service itself is the one receiving the request so the HTTP client will timeout)(?)

                            • 11. Re: Combining or Merging Indesign Files (IDML/INDT)
                              Loic.Aigon Adobe Community Professional

                              Dirk Becker and diggeralmu

                               

                              Ok well noted that merging idml outside of InDesign is all but a practical way of merging indesign documents.

                               

                              Loic

                              • 12. Re: Combining or Merging Indesign Files (IDML/INDT)
                                Dirk Becker Level 4

                                You might be interested in the thread here, recently revived by some Adobe engineer. I hope after my comment he understands the difference between "start" and "invoke".

                                 

                                In the meantime I've also bothered Google - apparently on Windows you can use sc.exe to start and stop a service on a remote machine: sc \\remotePC stop "Service_name"

                                There were also pointers towards PowerShell.

                                • 13. Re: Combining or Merging Indesign Files (IDML/INDT)
                                  absqua Level 4

                                  Couple things:

                                   

                                  1. Unless there's a reason you can't you should address spreads rather than pages. It will be much faster. Like: document.spreads.everyItem().duplicate(LocationOptions.AT_END, baseDocument)
                                  2. Maybe silly, but: you are definitely closing each document after you copy all of its pages/spreads into the base, right? Worth asking.
                                  3. If you can use .indt or .indd I don't know why you would bother with .idml in this application. It will be slower.

                                   

                                  Jeff