10 Replies Latest reply on Jul 21, 2015 10:20 PM by darrenten

    Memory Leak - targetengine?!

    TinoRuntzler Level 1

      Hi folks,

       

      I was analyzing some strange behavior of JavaScript concerning memory consumption, garbage collection. Following some questions, observations:

       

      1. Inner Functions/Objects/Function Pointers:


       

      function Hashtable() {

      this.valueList = new Array();

      var test = function() { return 1; }

      this.putValue = function(name, value) {

      this.valueList[name] = value;

      }

      }

       

      for (var i = 0; i < 100000; i++) {

      var hash = new Hashtable();

      hash.putValue("Key" + i, "Value" + i);

      delete hash;

      hash = null;

      }

       

      Oberservation: Running the script with the inner function test, the memory will rise per loop, despite the use of delete and null per loop. Because of the memory consumption the execution time will rise exponential. Without the inner function, the memory won't rise and the script is much faster.

       

      Question: What is so special on an inner function for the JavaScript engine? What kind of engine are you using, does it has a JIT compiler? Why can't I kill the object references with delete/setting null?

       

       

      2. Usage of targetengine directive:

       

      //Startup Scripts/script1.jsx

      #targetengine "my_special_target"

      function PersistentObject() {

           this.getXYZ = function() {...}

      }

      ...

      MyPersistendObject = new PersistentObject();

      ...


      //Scripts/script2.jsx

      #targetengine "my_special_target"

      var newObject = new Object();

      MyPersistendObject.getXYZ();

       

      Oberservation: Using this code from within a Startup Script, I can create peristend objects, I can use from a second script. The second script needs to "connect" to the same targetengine to call something like MyPersistendObject.getXYZ(). That's very usefull for caching etc. But all my code within the second script seems to be peristent as well. If I run the second script, all objects will be created twice and the references of the first run will reside too. This means the memory consumption will rise linear with the objects of second script per run. The execution time will rise exponential because of expensive memory allocation.

       

      Question: Does targetengine means no run of garbage collection at all? How can I override reference instead of creating new ones. Can I influence the garbage collection at all (with $.gc())? Is their a difference between the garbage collection of InDesign CS4 vs. InDesign Server  CS4?

       

      Thanks for any information.

       

      Tino

        • 1. Re: Memory Leak - targetengine?!
          Bob Stucky Adobe Employee

          The normal sequence of operation of an InDesign script, say launched from the Scripts Panel or the ESTK is:

           

          1. Create new scripting engine

          2. Execute the script

          3. Destroy the scripting engine.

           

          So when you are "normally" executing a script, the engine appears, runs, and disappears, which is why you can't have persistent objects/variables. After execution the engine is gone.

           

          With the #targetengine directive, you are specifically telling InDesign to execute the script in a persistent, named engine.

           

          Given a directive of:

           

          #targetengine "MyEngine"

           

          If the script is being executed as a startup script, and there is no engine named "MyEngine", InDesign will create a persistent engine named "MyEngine". Because that engine does not die until InDesign quits, anything that is done (e.g. creating a global persistent object) will remain accessible until InDesign quits. It is a fully functional engine that has all the features of any ExtendScript engine, just like any other.

           

          You are not "connecting" to another engine. You are executing in that engine. So your second script will be persisted.

           

          I am in no way qualified to talk about the way garbage collection is implemented in ExtendScript. But I do know that garbage collection is a background process. You can invoke it with $.gc(); but, there could be any number of qualifying conditions as to whether or not garbage is immediately collected. There could be other dependent background processes that have to execute before a variable is deemed collectable. I simply don't know.

           

          What I do know is that I have created InDesign projects that are multiple thousands of lines long. They create global variables that, during execution can temporarily grow into the tens of thousands (possibly more!) of values. One script that rendered html in an InDesign frame was a recursive beast that successfully laid out tables that were thousands of cells long, sometimes nested 4 or 5 tables deep (tables in table cells) maintaining references to all tables and formatting information throughout the process because the final formatting (specifically cell width calculations) had to happen at the end of each recursion branch. I was able to run the process to create 70 and 80 page documents, each taking an hour to generate. I could run these all day with no memory or garbage collection problems, and I never called $.gc().

           

          Bob

          • 2. Re: Memory Leak - targetengine?!
            Harbs. Level 6

            Tino,

             

            I believe there is a memory leak somewhere in the scripting engine. 

            I've spent time trying to pin down exactly where the problem is so I 

            could file a proper bug report. My feeling is that under certain 

            circumstances persistent variables/function do not get purged from 

            memory even when they should.

             

            There is definitely a memory problem where InDesign continuously uses 

            more memory which I first noticed in CS3. I would not exclude the 

            possibility that it is caused by the "scripted functions" of InDesign 

            (such as XHTML Export) which were introduced in CS3. This is all 

            conjecture, since I have not been able to pin down a clear cause.

             

            One repeatable problem can be shown by randomly opening and closing a 

            few documents (you don't even eed t do anything or save). The memory 

            usage will very gradually climb as the documents are opened and 

            closed. It will not be clear numbers -- as some of the memory WILL be 

            regained -- but not all.

             

            I hope the engineers will be able to use your simple example to get to 

            the bottom of this...

             

            Harbs

            • 3. Re: Memory Leak - targetengine?!
              Olav Martin Kvern Level 3

              Fellow Scripters,

               

              Could this possibly be related to InDesign's undo, rather than a memory leak in scripting? One way to test this possibility would be to run a test script using the app.doScript() method. Track memory use for each of the undo modes--if there's a difference between UndoModes.scriptRequest and UndoModes.fastEntireScript, then the problem is probably undo-related. If you see the same memory use growth in both scripts, then it's probably scripting-related.

               

              Thanks,

               

              Ole

              • 4. Re: Memory Leak - targetengine?!
                Marc Autret Level 4

                Hi TinoRuntzler,

                 

                My two cents about your question #1.

                 

                a) In your loop the statement:

                 

                delete hash;

                 

                won't have any effect, because hash is a declared variable.

                (See http://perfectionkills.com/understanding-delete/)

                 

                You can verify this fact by adding an alert:

                ...

                var hash = new Hashtable();

                hash.putValue("Key" + i, "Value" + i);

                delete hash;
                alert( hash.valueList.__count__ );
                  // => 1

                break;

                ...

                 

                Thus, hash still exists and contains data at this point.

                 

                b) Anyway, you probably don't need deletion in that context. The statement:

                 

                hash = null;

                 

                works fine and should facilitate garbage collection regarding hash.

                 

                c) Nevertheless, you still have potential memory leak because of the test inner function (=closure) within the Hashtable constructor. If you add:

                 

                alert( $.summary( ) );

                 

                at the end of your code, you'll find that a huge number of Function and (workspace) entries are still registered, despite hash = null.

                 

                Actually, each time you call the Hashtable constructor —and you call it 100,000 times—, JS creates a new test function in its own workspace in order to allow the closure's mechanism. It seems that nullifying hash later has no effect on this.

                 

                That's why you probably have to define an explicit destructor:

                 

                this.destroy = function() {test = null;}

                 

                and call it from your loop to clean up memory:

                 

                function Hashtable()
                     {
                     this.valueList = new Array();
                
                     var test = function() { return 1; };
                
                     this.putValue = function(name, value)
                          {
                          this.valueList[name] = value;
                          };
                
                     this.destroy = function(){test=null;};
                     }
                
                var hash, i;
                for( i=0 ; i < 100000; ++i )
                     {
                     hash = new Hashtable();
                     hash.putValue("Key" + i, "Value" + i);
                
                     hash.destroy(); // call the destructor
                
                     //delete hash;
                     hash = null;
                     }
                
                alert($.summary());
                

                 

                @+

                Marc

                1 person found this helpful
                • 5. Re: Memory Leak - targetengine?!
                  Harbs. Level 6

                  Hi Marc,

                   

                  Excellent analysis!

                   

                  Moral of the story: avoid closures whenever possible!

                   

                  Harbs

                  • 6. Re: Memory Leak - targetengine?!
                    Harbs. Level 6

                    Wouldn't is be simpler to make test a method of Hashtable?

                     

                    this.test = function (){return 1;}
                    

                     

                    Of course this would make it public, but I'm not convinced that private vs. public is such an issue in JS, and it's a small price to pay to avoid having to call destroy()...

                     

                    Harbs

                    • 7. Re: Memory Leak - targetengine?!
                      TinoRuntzler Level 1

                      Hi Harbs and Marc,

                       

                      thanks for your posts.

                       

                      Actually we got rid of the OO pattern of information hiding and using closures to express private acess/function because of the issues you've shown.

                       

                      I'm a bit sad about it, cause i like to describe, express programmatically how to use the code. Now we are using the <object_name>.prototype.<function_name> signature to define all methods on an instantiable object. We don't differentiate between public and private access anymore.

                       

                      The leads to a better performance but it forces the developer to describe and mark public and private methods in their comments. So you have to be carefully by writing comments an annotating your code.

                       

                      Cheers Tino

                      • 8. Re: Memory Leak - targetengine?!
                        Harbs. Level 6

                        Another way to go about it would be to use global helper "classes".

                         

                        That way, all methods of an object would be public, and for anything "private" you simply use the helper function instead.

                         

                        Harbs

                        • 9. Re: Memory Leak - targetengine?!
                          Dirk Becker  Level 4

                          Marc,

                           

                          interesting that this thread is revived after such a long time. I already convinced Tino with performance and memory statistics, actually around the time when you brought up closures on Twitter.

                           

                          Harbs,

                           

                          a large share of recent additions in our project has already advanced in the direction of helper classes. We just call them XyzUtils for basic functionality that not even requires real objects beyond the namespace, or Facade for the higher level entry functions. I try to loosely follow the simplest patterns from the plugin SDK here.

                           

                          Note that not all leaks are caused by closures. You can introduce circular dependencies with prototypes that also fail in garbage collection. The only chance to find them is permanent watching of $.summary() .

                           

                          Dirk

                          • 10. Re: Memory Leak - targetengine?!
                            darrenten Level 1

                            Hello,

                             

                            I faced that problem while tring to export indd file to pdf with a jsx script in InDesignServer CC 2014. The memory keep climbing!

                            What's the problem?

                             

                            var result = "No documents are open................";

                            // var argsNames=app.scriptArgs.getValue("Name").split(";")

                            argsNames = arguments;

                             

                             

                            var fileName = argsNames[0];

                            var argPreset;

                            var storeLocation;

                            if (argsNames && argsNames[1]) {

                                argPreset = argsNames[1];

                            }

                            if (argsNames && argsNames[2]) {

                                storeLocation = argsNames[2];

                            }

                            argsNames=null;//clean

                             

                             

                            Log("Params:" + fileName + ";" + argPreset + ";" + storeLocation)

                            //OpenOptions.DEFAULT_VALUE

                            //OpenOptions.OPEN_ORIGINAL

                            //OpenOptions.OPEN_COPY

                            var myDocument = app.open(new File(fileName),OpenOptions.OPEN_COPY);

                            //Fill in a valid file path on your system.

                            var myFolder = new Folder(storeLocation);

                            result =($.summary());

                            result += "saved to folder:  " + myFolder.fullName;

                            if (!myFolder.exists && !myFolder.create()) {

                                result = "Not saved.  Unable to create the folder:  " + myFile.parent.fullName;

                            } else {

                                //myExport2SepratedPages(myDocument, myFolder.fullName, argPreset);

                                myExportAllPages(myDocument, myFolder.fullName, argPreset);

                            }

                            myFolder=null;//clean

                            //myDocument.close();

                            myDocument = null;//clean

                            result+=($.summary());

                            $.gc();

                            result;

                            function myExport2SepratedPages(myDocument, myFolderName, presetFile) {

                                var myBaseName = myDocument.name;

                                for (var myCounter = 0; myCounter < myDocument.pages.length; myCounter++) {

                                    myPageName = myDocument.pages.item(myCounter).name;

                                    app.pdfExportPreferences.pageRange = myPageName;

                                    app.importFile(ExportPresetFormat.DOCUMENT_PRESETS_FORMAT, presetFile);

                                    //Generate a file path from the folder name, the base document name, and the page name.

                                    //Replace the colons in the page name (e.g., "Sec1:1") wtih underscores.

                                    var myRegExp = new RegExp(":", "gi");

                                    myPageName = myPageName.replace(myRegExp, "_");

                                    myFilePath = myFolderName + "/" + myBaseName + "_" + myPageName + ".pdf";

                                    myFile = new File(myFilePath);

                             

                                    myDocument.exportFile(ExportFormat.pdfType, myFile);

                              myDocument.close(SaveOptions.NO);

                              myDocument = null;//delete document

                             

                                }

                            }