10 Replies Latest reply on Dec 7, 2011 6:02 AM by Harbs.

    How to Cure Large Memory Consumption

    mlavie Level 1

      For CS5, Windows 7 64 Bit, ActionScript

       

      I have written an Extension which analyzes a data feed and build from it a large dicument consisting of tens of pages - each page with perhaps 100 PageItems.

       

      When I try to produce the whole document at one time, it usually fails after about 15-20 pages, depending on the number of PageItems on each page. I also notice that the InDesign.exe process consumes about 1.9 GB when the explosion occurs. The "explosion" is "stupid" error messages, such as a null exception error, when in fact the data in question clearly has a value, and other strange things.

       

      I have noticed that Tables are paricularly troublesome - tables with many rows - and particularly many columns seem to aggravate the situation.

       

      Closing the Extension itself releases about 160 MB.

       

      In addition, a tremendous amount of memory seems to be taken by each page in the Document. When I close the document, it frees up about 30 MB per page. Since I have well over 30 pages in many documents, this is obviously an issue.

       

      I have taken the following precautions:

       

      1) I perform a save of the document after every page

      2) I force Garbage Collection after every page

      3) To the best of my knowledge, I set all my objects (including DOM objects such as TextFrame, etc.) to null as soon as I am done with them.

       

      There are some things which are not clear to me. For example, if I defined an Array of Table Columns, do I need to set each entry in the Array to null when I'm done, even if i nullifed the Table object from which they were taken?

       

      I would be gratefull for any advice.

       

      TIA,

      mlavie

        • 1. Re: How to Cure Large Memory Consumption
          Harbs. Level 6

          Welcome to the joys of InDesign scripting...

           

          Yes. Tables are particularly problematic. Too much text processing can cause memory bloat as well.

           

          There's many tricks to work around some of these issues, but I don't even know where to start (especially without seeing the code)...

           

          FWIW, None of the steps you've taken will help very much except for possibly saving the document (but only if you do a "Save As").

           

          Harbs

          • 2. Re: How to Cure Large Memory Consumption
            Marc Autret Level 4

            mlavie wrote:

             

            […]

            I have noticed that Tables are paricularly troublesome - tables with many rows - and particularly many columns seem to aggravate the situation.

            […]

             

             

            I've experienced similar over-heating in my IndexMatic script when it needs to traverse table contents. InDesign tables are surprisingly heavy!

             

            More generally, fighting against memory consumption in a script is a kind of black magic that involves two distinct levels:

            1. dealing with specific JS / ExtendScript memory leaks (removing references, nullifying closure-related vars, garbage collection, etc.)

            2. dealing with specific DOM / ID command processing issues during a huge process (undo modes, 'redraw', periodic backup of the doc in "Save as" mode, etc.)

             

            Level 2 strategies are fairly empirical and depend heavily on the scope of the process. I confess I'm unable to make a list of good practices on this subject—and I'd love to find one.

             

            In my opinion, level 1 issues don't impact much the execution time and the memory consumption. However, it's probably better to help the GC to do the housework, especially if you use a persistent target engine. I usually nullify object references, including DOM specifiers, when they become unused or invalid. Also, I find it healthy to explicitly destroy custom objects using a deep "delete obj.prop" on each owned property before setting the obj to null. The usual way to destroy an Array is: myArray.length=0, then myArray=null. When you watch a property through obj.watch(…), don't forget to unwatch it. In the ScriptUI area, there are more specific issues involving closures and event listeners. I recommend to always call the appropriate removeEventListener methods before destroying a widget (or the whole window). There would also be much to say about cleaning function references in that context.

             

            Finally, ExtendScript provides a very useful code to identify residual references at the end of an entire process:

            alert( $.summary() );

             

            @+

            Marc

            • 3. Re: How to Cure Large Memory Consumption
              John Hawkinson Level 5

              Marc:

              I usually nullify object references, including DOM specifiers, when they become unused or invalid. Also, I find it healthy to explicitly destroy custom objects using a deep "delete obj.prop" on each owned property before setting the obj to null. The usual way to destroy an Array is: myArray.length=0, then myArray=null.

              Why would you do this? It's basically saying you don't trust the language's garbage collection to do its job. Wouldn't it  be best to simply structure your program so that the variables are appropraitely scoped and are necessarily destroyed when they go out of scope?

              Function references and listeners are different of course, but what you describe doesn't sound like it is healthy -- it sounds like it is a waste of effort and makes a clean looking program look ugly.

               

              Can you give an example where there is any point to this?

              • 4. Re: How to Cure Large Memory Consumption
                mlavie Level 1

                First of all, thanks to Marc and Harbs for the quick responses.

                 

                I have the following follow-up questions:

                 

                1) Both of you said make sure to use SaveAs (and not Save). Must I give the document a new name, or can the SaveAs save it with its current name (in order to gain the memory release benefit, of course).

                2) I use "System.gc()" in order to request garbage collection. Is there a better way? Is there something additional I must do?

                3) How can I specify not to support "undo" and "redraw" in ActionScript (I assume that Marc meant that "undo" and "redraw" are memory hogs).

                4) Is there an ActionScript method which is the equivalent of $.summary() ?

                5) If I use the jsx interface to call $.summary(), will it still return usefull results?

                6) Do I need to to delete EVERY property in my DOM objects? For example, if I have created a Table object, must I delete all 99999 properties before nullifying it? If one of thise properties is a pointer to another object (Table points to its Stroy's insertion point, which is itself an InsertionPoint object), must I nullify the Table.storyInsertionPoint property deleting it?

                 

                TIA,

                mlavie

                • 5. Re: How to Cure Large Memory Consumption
                  mlavie Level 1

                  Harbs/Marc,

                   

                  As regards point 1 in my response above:

                   

                  I see that there is no SaveAs, but there is a SaveACopy. Is that what you were referring to? And if so, does the original document then take up less memory? And so as not to leave dummy files around uneccessarily, can I do a SaveACopy to the original name?

                   

                  TIA,

                  mlavie

                  • 6. Re: How to Cure Large Memory Consumption
                    Harbs. Level 6

                    Supply the document file to save() to do a save as. You don't need to give it a new name.

                     

                    In my opinion calling gc is a waste of time as is trying to nullify javascript objects.

                     

                    I'm pretty sure that the vast majority of the memory bloat is caused by InDesign proxy objects which has nothing to do with the javascript garbage collection. The trick is to try to prevent the build up of proxy objects which is no small feat...

                    • 7. Re: How to Cure Large Memory Consumption
                      Marc Autret Level 4

                      Hi John,

                      John Hawkinson wrote:

                       

                      […] Why would you do this? It's basically saying you don't trust the language's garbage collection to do its job. […]

                       

                      Yes, that's exactly my point. I really do not trust the ExtendScript garbage collection. I've experienced indisputable memory leaks in CS3 and CS4, especially when ScriptUI is involved. A few months ago, Dirk Becker advised me to call $.gc() twice in some circumstances—and this worked! I have a script, Wordalizer, which uses a huge number of Array objects in order to compute path points within a quad tree. The current release seems really clean to me and relies on the default memory management. But it happens that Win64 platform users experience InDesign issues, and need to restart the app, when they launch the script several times during the same session. My conclusion is: when working on a huge project that uses a lot of temporary objects, closures, ScriptUI event listeners, etc., better is to facilitate garbage collection.

                       

                      John Hawkinson wrote:

                       

                      […] but what you describe doesn't sound like it is healthy -- it sounds like it is a waste of effort and makes a clean looking program look ugly. […]

                       

                      Yes, that's true. And I have no simple example where there is any point to do what I suggest. Actually, this only can make sense when one deals with memory consumption issues ;-)

                       

                      @+

                      Marc

                      • 8. Re: How to Cure Large Memory Consumption
                        mlavie Level 1

                        Just a reminder - I'm talking about ActionScript...

                         

                        Is it the implication of Harbs/Marc's comments that nullification of Proxy objects (such as TextFrame, table, etc) or delete-ing all their properties is useless? I ask because nullifying all my |textFrames, tables, etc., doesn't help.

                         

                        mlavie

                        • 9. Re: How to Cure Large Memory Consumption
                          John Hawkinson Level 5

                          mlavie:

                          Just a reminder - I'm talking about ActionScript...

                          I think we all forgot that.

                           

                          Is it the implication of Harbs/Marc's comments that nullification of Proxy objects (such as TextFrame, table, etc) or delete-ing all their properties is useless? I ask because nullifying all my |textFrames, tables, etc., doesn't help.

                          I think you've misunderstood what a proxy object is -- it is an object that InDesign creates and reference-counts in order to support accessing the object via scripting. It is not the "object" or reference thereto available in JS or AS3.

                           

                          With respect to nullification/deletion of properties, it certainly should be useless, and I think the answer is that it is useless except for some bizarre bugs. And those are all problems with the ExtendScript interpretter and one should not expect them to carry over to AS3.

                           

                          Fundamentally, I think you need to do more tests, rather than asking us. I mean, asking is good, but this stuff is finicky enough that your milage may vary, and doing tests is an excellent way to educate yourself.

                          • 10. Re: How to Cure Large Memory Consumption
                            Harbs. Level 6

                            John Hawkinson wrote:

                             

                            I think you've misunderstood what a proxy object is -- it is an object that InDesign creates and reference-counts in order to support accessing the object via scripting. It is not the "object" or reference thereto available in JS or AS3.

                             

                            That's right. They are C++ objects, and nothing you can do in ES or AS will make the slightest bit of difference. The only way of adressing those problems are to prevent the creation of the proxy objects in the first place...