16 Replies Latest reply on Sep 8, 2015 4:01 PM by Qwertyfly...

    Speed and Stability in batch processing of files.

    elDudereno Level 3

      My company is changing some aspects of our setup files. We have THOUSANDS. I have written a working script to work through all of them.

       

      SPEED:

      When I start the script I'll get as many as 10 per minute, but as the batch goes on it gets slower and slower. I let it run over night and it was down to one file every three minutes.

      Right now my bandaid for this is to run 50 files at a time.

       

      STABILITY:

      When the script is finished I have to force quit illustrator because it is locked up.

       

      I'm running IllustratorCC2015 on a mac OS X 10.10.5

      I've wrapped the whole script in a function, and added $.gc(); garbage collection commands because of some tips I found here on the forum. Any advice on either issue would be most welcome. Here is my code.

       

      // Main Code [Execution of script begins here]
      regRemv();
      function regRemv(){
        // uncomment to suppress Illustrator warning dialogs
        // app.userInteractionLevel = UserInteractionLevel.DONTDISPLAYALERTS;
      
        // establish save options
        var saveOptions = new IllustratorSaveOptions();
        saveOptions.compatibility = Compatibility.ILLUSTRATOR15;
      
        var destFolder, sourceFolder, files, fileType, sourceDoc, targetFile;
        // Select the source folder.
        sourceFolder = Folder.selectDialog( 'Select the folder with Illustrator files you want to process');
        // If a valid folder is selected
        if ( sourceFolder != null ){
        files = new Array();
        fileType = prompt( 'Select type of Illustrator files to you want to process. Eg: *.ai, *.eps', '*' );
        // Get all files matching the pattern
        files = sourceFolder.getFiles( fileType );
        if ( files.length > 0 ){
        // Get the destination to save the files
        //destFolder = Folder.selectDialog( 'Select the folder where you want to save the converted PDF files.', '~' );
        destFolder = sourceFolder;
        for ( h = 0; h < files.length; h++ ){
        sourceDoc = app.open(files[h]); // returns the document object
        var idoc = sourceDoc;
        var straypoints = [];
        for (n=0; n<idoc.pathItems.length; n++) {
        var ipath = idoc.pathItems[n];
        if (ipath.pathPoints.length==1)
        straypoints.push(ipath);
        }
        for (m=0; m<idoc.textFrames.length; m++) {
        var itext = idoc.textFrames[m];
        if (itext.textRange.length==0)
        straypoints.push(itext);
        }
        for (k=0; k<straypoints.length; k++){
        straypoints[k].remove();
        }
      
        var pi = idoc.pathItems;
        var tf = idoc.textFrames;
        createGuideDie()
        var sw = idoc.swatches['Guide-die'];
      
        // text frames with registration color change to die color.
        for (var t=tf.length -1; t>=0; t--) {
        var iTxtFrm = tf[t];
        var firstLtr = iTxtFrm.characters[0];
        var chars = iTxtFrm.characters;
        if (iTxtFrm.layer.visible === true && iTxtFrm.layer.locked === false && iTxtFrm.hidden === false && iTxtFrm.locked === false && firstLtr.characterAttributes.fillColor.typename === "SpotColor" && firstLtr.characterAttributes.fillColor.spot.name === "[Registration]"){
        for (var c=0; c<chars.length; c++) {
        var ltr = chars[c];
        ltr.characterAttributes.fillColor = sw.color;
        }
        }
        }
      
        for (var i=pi.length -1; i>=0; i--) {
        var ipath = pi[i];
        if (ipath.layer.visible === true && ipath.layer.locked === false && ipath.hidden === false && ipath.locked === false && ipath.stroked === true && ipath.strokeColor.typename === "SpotColor" && ipath.strokeColor.spot.name === "[Registration]"){
        // ipath.strokeColor = sw.color;
        ipath.remove();
        // ipath.stroked = false;
        } else if (ipath.layer.visible === true && ipath.layer.locked === false && ipath.hidden === false && ipath.locked === false && ipath.stroked === false && ipath.filled === false && Math.round(ipath.height) === 18 && Math.round(ipath.width) === 18){
        ipath.remove()
        }
        }
      
        var here = sourceDoc.fullName;
        sourceDoc.saveAs( here, saveOptions );
        sourceDoc.close();
        $.gc();
        $.gc();
        }
        }
        } else {
        alert( 'No matching files found' );
        }
        $.gc();
        // app.executeMenuCommand('closeAll');
      
        app.beep();
        alert(h + " finished processing");
      
      
        function createGuideDie(){
        if (!doesColorExist('Guide-die')){
        // Create CMYKColor
        var cmykColor = new CMYKColor();
        cmykColor.cyan = 28;
        cmykColor.magenta = 56;
        cmykColor.yellow = 0;
        cmykColor.black = 37;
        // Create Spot
        var spot = idoc.spots.add();
        spot.color = cmykColor;
        spot.colorType = ColorModel.SPOT;
        spot.name = 'Guide-die';
        }
        }
        function doesColorExist(){
        var clrs = idoc.swatches;
        for (var i=0;i<clrs.length;i++){
        if (clrs[i].name==='Guide-die'){
        return true;
        }
        }
        return false;
        }
      }
      $.gc();
      
      
        • 1. Re: Speed and Stability in batch processing of files.
          Silly-V Adobe Community Professional

          I sometimes have the same problem, and would also like to know what causes it and how to deal with it, in CC2015.

          • 2. Re: Speed and Stability in batch processing of files.
            W_J_T Level 4
            elDudereno: We have THOUSANDS. I have written a working script to work through all of them.


            It seems to be related to memory leak/space issues, I have seen the same thing (slow, crawl, crash) mentioned regarding photoshop, after effects, indesign, etc., batch scripts as well, dating back well before CC...

            As you well know from you own previous thread: batch script memory use CS5.1 vs CS6


            ----------------------------------------------------------------------------------------------------------------------------------------------

            From the Illustrator Scripting Guide:

             

            Known issues:

             

            Scripts that create, save, and close many Illustrator files should periodically quit and relaunch Illustrator. The recommended maximum number of files to process before quitting and relaunching Illustrator is:

             

            • Windows 500 files

            • Mac OS 1000 files

             

            The “An Illustrator error occurred: 1346458189 (“PARM”)” alert may be popped when badly written scripts are repeatedly run in Illustrator from the ESTK.

             

            Scripters need to be very careful about variable initialization and namespace conflict when pushing a batch of Illustrator scripts repeatedly for execution in Illustrator via the ESTK in one Illustrator session. Each script run is executed within the same persistent ExtendScript engine within Illustrator.

             

            ^ That was copied pasted from the CS5 version, amazing they cant seem to fix/improve something that has been known for so long. The same verbiage is in the CS3 Guide from 8+ years ago.

             

            Example Topic: https://forums.adobe.com/message/3254831


            Post 16 - CarlosCanto May 13, 2013

             

            I think I read somewhere that running a script a bunch of times causes the ESTK to crawl, it has a limit. Restart both the ESTK and Illustrator and try again, or also try running the script direct from within Illustrator... I may be wrong though.

            Post 17 - Muppet Mark May 14, 2013


            The guide also suggests a max of 1000 files on mac and 500 on pc if you are performing a batch routine…

            ----------------------------------------------------------------------------------------------------------------------------------------------

            elDudereno: added $.gc(); garbage collection commands

             

            You might also research about using: null, undefined, delete for your variables and functions once they are utilized to further clear/clean things up manually. However you still probably wont get around the file count limits mentioned above.

            • 3. Re: Speed and Stability in batch processing of files.
              elDudereno Level 3

              I was able to get my Maximum number of files per minute up to 13 at the beginning by adding this between lines 73 & 74.


              // clean up
              sourceDoc = null;
              idoc = null;
              straypoints = null;
              n = null;
              m = null;
              pi = null;
              tf = null;
              sw = null;
              t = null;
              iTxtFrm = null;
              firstLtr = null;
              chars = null;
              c = null;
              ltr = null;
              ipath = null;
              here = null;
              

              I'm sure some of this is redundant due to scope and whatnot, but I don't have a full grasp on that yet so I thought I'd be thorough.


              But in next minute it's less and by the time I've run the 50 files the last full minute is only 4 files.


              Illustrator is still locked up and I have to force quit.

              • 4. Re: Speed and Stability in batch processing of files.
                Silly-V Adobe Community Professional

                Wow hey, what if you try to set ipath to null, every time you have an ipath.remove() ?
                Will that make it even faster?

                • 5. Re: Speed and Stability in batch processing of files.
                  elDudereno Level 3

                  That got my max up to 15 in the first minute.

                   

                  I processed

                  15 files in 2:15

                  8 files in 2:16

                  7 files in 2:17

                  6 files in 2:18

                  5 files in 2:19

                  4 files in 2:20

                  5 files in 2:21

                  4 files in 2:22

                   

                  54 files in 7 minutes is pretty great, but I don't understand why it slows down so quickly. I would really rather have the issue be with my code and NOT with Adobe. I can improve my code. It's frustrating that a script that could potentially process 900 files an hour has to be babysat all day and only does about 250 an hour because of the slow clumsy human (me).

                   

                  Does anyone have any insight into the program being locked up at the end of the script? Is there a nicer way to close out my files?

                  • 6. Re: Speed and Stability in batch processing of files.
                    Silly-V Adobe Community Professional

                    In one of my scripts, I was able to change it from opening a new file every time for processing of certain stuff, to removing all the pageItems in my document and re-using the same document to process the next set. However, for scripts where files must actually be opened, and there really is no solution, or using platform-specific languages also fails, I'd then resort to trying to use Actions with a batch process to do an instance of the script for each of the batch-processed files.

                     

                    I do notice that on CC2015, sometimes it would spin for a long time after finishing execution. I've been lucky that so far, it's been tolerated by people.

                    • 7. Re: Speed and Stability in batch processing of files.
                      CarlosCanto Adobe Community Professional & MVP

                      I agree with all of the above, it must be memory management on their end, nothing much we can do but to find clever ways to make our code run close to optimal for a few times, before it goes back to crawling.

                      • 8. Re: Speed and Stability in batch processing of files.
                        Qwertyfly... Level 4

                        i did a re write on your script.

                         

                        not sure if its any better but would be interested to see if it runs any faster.

                        only done a few quick test so double check it does the right thing to your files.

                         

                        and make sure you have backups as I am not sure how well it will work.

                         

                        let me know how it goes.

                         

                        function regRemv(){  
                            // save options  
                            var saveOptions = new IllustratorSaveOptions();  
                            saveOptions.compatibility = Compatibility.ILLUSTRATOR15;  
                            var myString;
                            //get Files
                            var files = Folder.selectDialog( 'Select the folder with Illustrator files you want to process').getFiles(prompt( 'Select type of Illustrator files to you want to process. Eg: *.ai, *.eps', '*' ));
                            if(files.length == 0){alert( 'No matching files found' ); return;}
                            //Loop Files
                            for(var d = 0; d < files.length; d++){
                                var doc = app.open(files[d]); //Open file
                                var item = doc.pageItems;
                                var col = getGuideDie() // make sure 'guide-die' exists
                                for(var i = item.length-1; i > 0; i--){
                                    if((item[i].typename == 'PathItem' && item[i].pathPoints.length == 1)||(item[i].typename == 'TextFrame' && item[i].textRange.length == 0)){
                                        item[i].remove(); // Remove StrayPoints
                                    }else if(item[i].layer.visible === true && item[i].layer.locked === false && item[i].hidden === false && item[i].locked === false){
                                        if(item[i].typename == 'TextFrame'){// Cycle all chars and change from 'registration' to 'guide-die'
                                            for(var c = 0; c < item[i].characters.length; c++){
                                                if(item[i].characters[c].characterAttributes.fillColor.typename === "SpotColor" && item[i].characters[c].characterAttributes.fillColor.spot.name === "[Registration]"){ 
                                                    item[i].characters[c].characterAttributes.fillColor = col.color;
                                                }
                                            }
                                        }else if(item[i].typename == 'PathItem' ){
                                            if(item[i].stroked === true && item[i].strokeColor.spot.name === "[Registration]"){
                                                item[i].remove(); // if pathItem color is 'registration' delete
                                            }else if(item[i].stroked === false && item[i].filled === false && Math.round(item[i].height) === 18 && Math.round(item[i].width) === 18){
                                                item[i].remove(); // if pathItem is invisable and 18pt x 18pt, delete
                                            }
                                        }
                                    }
                                }
                                doc.saveAs(doc.fullName,saveOptions);
                                myString = myString + doc.name + " - Processed\n";
                                $.writeln(doc.name);
                                doc.close(SaveOptions.DONOTSAVECHANGES);
                            }
                            alert(myString);
                            
                            function getGuideDie(){ 
                                try{
                                    return doc.swatches.getByName('Guide-die');
                                }catch(e){
                                    var cmykColor = new CMYKColor();  
                                    cmykColor.cyan = 28;  
                                    cmykColor.magenta = 56;  
                                    cmykColor.yellow = 0;  
                                    cmykColor.black = 37; 
                                    var spot = idoc.spots.add();  
                                    spot.color = cmykColor;  
                                    spot.colorType = ColorModel.SPOT;  
                                    spot.name = 'Guide-die';
                                    return spot;
                                }
                            }
                        }
                        regRemv();
                        
                        • 9. Re: Speed and Stability in batch processing of files.
                          elDudereno Level 3

                          Thanks Qwertyfly!

                           

                          I tried your script (line 49 needs idoc changed to doc). I started the script at 7:22:55 trying to catch the beginning of the next minute.

                           

                          I processed:

                          1 files in 7:22

                          13 files in 7:23

                          9 files in 7:24

                          6 files in 7:25

                          6 files in 7:26

                          5 files in 7:27

                          5 files in 7:28

                          4 files in 7:29

                          4 files in 7:30

                           

                          The program is locked up and has to be force quit & restarted. 53 files in 8 minutes. I see that you didn't include any garbage collection or neutralizing of variables, was the theory that the more elegant execution wouldn't require it? (your code is beautiful by the way.)

                           

                          Adobe still seems responsible for the slowdown, unless there is something we are all missing. With the level of super talented contributors this thread is getting, I doubt it.

                           

                           

                           

                          P.S. I tried adding some garbage collection and clean up of variables to your script and with no noticeable improvement. (edited at 8:21 am)

                          • 10. Re: Speed and Stability in batch processing of files.
                            elDudereno Level 3

                            I noticed that there were no executeMenuComand lines in the script, so I ran it in Illustrator CS5.1 as an experiment. (that's the only other version I have) It opened the ExtendScript Toolkit.

                             

                            I processed:

                            4 files in 8:31

                            26 files in 8:32

                            60 files in 8:33

                            36 files in 8:34 (there was a popup about a file from a newer version of Illustrator)

                            63 files in 8:35

                            33 files in 8:36

                             

                            So thats 222 files in 6 minutes.... It didn't need to be force quit.

                             

                            Yep.

                             

                            The answer is to use CS5.1.

                             

                            I've submitted a bug report.

                            • 11. Re: Speed and Stability in batch processing of files.
                              W_J_T Level 4

                              elDudereno wrote:

                               

                              I ran it in Illustrator CS5.1 as an experiment. So thats 222 files in 6 minutes, It didn't need to be force quit. The answer is to use CS5.1.

                               

                              It seems like you have sadly come full circle with your previous thread (which I referenced above) batch script memory use CS5.1 vs CS6 "Are things any better in CC?" <-- it appears that the answer is a definitive "No" !

                              • 12. Re: Speed and Stability in batch processing of files.
                                Qwertyfly... Level 4

                                My thought was that less for loops and as few if statements as possible would help speed things up.

                                there are also a lot less variables. so the hope that it was not going to flog the js engine so much.

                                 

                                I created a simple test file and made 102 copies of it.

                                it ran through those in less then 3 mins.

                                 

                                i must have hit ctrl+z in the ESTK thinking I had illustrator active. thats why the idoc got back in there...oops.

                                 

                                I have never used $.gc(). and in the quick search I did I did not find much on it so left it out as I am not sure how it works exactly.

                                I know that browsers do garbage cleanup automatically, of course illustrator would make you do it manually...

                                I do clear variables in some situations, but as the only var that contains a large volume is the myString that accumulates the whole way through the script I thought it would not be worth the added processor cycles.

                                would be interesting to know if adding some var clears stabilizes the script at all.

                                 

                                I have found that CC+ has many more issues with the scripting engine. many more crashes these days...

                                • 13. Re: Speed and Stability in batch processing of files.
                                  Silly-V Adobe Community Professional

                                  Qwertyfly... wrote:

                                  I created a simple test file and made 102 copies of it.

                                  it ran through those in less then 3 mins.

                                  So are you saying that you are not experiencing the terrible slowdown on your machine? (I have not tested this code myself) - But have had reports that this happens on CC on Macs.

                                  • 14. Re: Speed and Stability in batch processing of files.
                                    Qwertyfly... Level 4

                                    I did not experience any slow down.

                                    granted the file I tested on was basic and had just enough elements to check each of the tasks the script was set to achieve.

                                     

                                    I found it actually got quicker.

                                    it started opening each file. flickering it as it did its thing then closing the file but then illustrator took on a "not responding" look,

                                    at that point it just took off.

                                    the first ten or so files soaked up most of the first min. then it stopped showing me the working and the rest went much faster.

                                     

                                    that raises an interesting thought.

                                    When programming Excel or the like you have the option of turning off screen updating.

                                    it not only looks cleaner and more professional but speeds up run times as well. ( I tested this years ago )

                                    it would be nice to have an option like that for illustrator.

                                    • 15. Re: Speed and Stability in batch processing of files.
                                      elDudereno Level 3

                                      And this was in CC2015? Was it on a Mac?

                                       

                                      The result you are describing is similar what I'm getting using CS5.1.

                                      • 16. Re: Speed and Stability in batch processing of files.
                                        Qwertyfly... Level 4

                                        cc2015, on a pc.

                                        an i7 processor with 24GB ram.

                                         

                                        granted I did crash illustrator 5 min before I ran the test so the Main JS engine was freshly flushed.

                                         

                                        I have found the js engine is less stable in the last few versions then it was in cs5 or cs6.