7 Replies Latest reply on Feb 23, 2017 2:27 AM by marque-1968

    45 second AE with expressions, takes 2 hours to render

    marque1968

      I make a sports program and I have the hourly standings to show. Last year I relied heavily on scripting to make the AE. However, this year I lessened the things that scripting does to mainly copy my template and populate a text-field with all the info I want to display.

       

       

      Through expressions the standings pages are created. Reading the info from the appropriate line in the text-field, and use the comma-sepatated info on that line to extract the name for that particular field. When it is for instance a country flag to be shown, then that is done with searching the correct frame where the flag is shown, and hold that frame for the duration of that 'page'. Other values are for instance taken from an overall settings composition, where info like the nr of the race and the series is being set, which in turn decides the correct colors to use and the event-graphics.

       

      I have about 350 lines in the standings-AE, including control layers (null layers) and masks.

       

      The rendering of each AE is too heavy, taking 2 hours each (7 to run). I am aware that this is due to the fact that all data in all the 350 'layers' is re-evaluated for each frame, and perhaps also for the default settings. However, I do not know how I can populate the values only once per AE (or at least, only the times it needs to change, like on showing a 'new page').

       

      Is there a way to speed it up? I don't want to lose the ability to change a small error in the text-field, and have a correct output, nor do I want to miss my general settings.

       

      Thanks for sharing your knowlede

       


      Marque.

       

      [Moved from non-technical Forum Lounge to specific Program forum... Mod]

      [Here is the list of all Adobe forums... https://forums.adobe.com/welcome]

        • 1. Re: 45 second AE with expressions, takes 2 hours to render
          Mylenium Most Valuable Participant

          Nothing to do with the expressions. I can run about 5000 lines of code on a 100 layers reasonably fast. You are obviously using time-remapping and valueAtTime() a lot and generally AE has a bottleneck with these temporal evaluations in terms of actually looking up the frame and values there. So quite likely the flaw is in your project structure to begin with and your code is merely a crutch to that end. In any case, without actually seeing your stuff, nobody can advise, but I strongly suspect that some refactoring from the ground up would mightily improve things. Or short: You need to start over with a clear mind and ditch your flawed approach.

           

          Mylenium

          • 2. Re: 45 second AE with expressions, takes 2 hours to render
            marque1968 Level 1

            Ok, let me try to make a recap of the expressions I have:

             

             

            There are 3 boxes for the sub catagories shown. As there can be up to 9 different sub catagories to be shown, there are 7 ways these three boxes must react. Therefore there are 7 null-layers, with markers set in the slider to the time when changes need to occur. There is also an "Compositing Option -> Effect Opacity" set. This determines the fade out and fade in of the boxes and the data:

            n = 0;

            spd = effect("Slider Control")("Slider");

            nk = spd.numKeys;

            if (nk > 0){

              n = spd.nearestKey(time).index;

              keyTime = spd.key(n).time;

              preLength = comp("_GeneralSettings").layer("vars-Standings").effect("sub-fade-pre")("Slider")*(1/25)

              postLength = comp("_GeneralSettings").layer("vars-Standings").effect("sub-fade-post")("Slider")*(1/25)

              if (keyTime >= time) {

              difftime = (keyTime - time);

              if (difftime <= preLength) {

               difftime / preLength * 100;

              } else { 100 }

              } else {

              difftime = (time - keyTime);

              if (difftime <= postLength) {

               difftime / postLength * 100;

              } else { 100 }

              }

             

            } else { 100 }

             

             

             

             

             

             

             

             

            To decide which of the 7 options, need to be chosen, there is a main NULL for the subselects. That slider is populated through:

            nrOfSubs = parseInt(thisComp.layer("_Info").text.sourceText.split("\r")[2])

            if (nrOfSubs <= 3) {

              thisComp.layer("subselect3").effect("Slider Control")("Slider")

            }

            else if (nrOfSubs == 4) {

              thisComp.layer("subselect4").effect("Slider Control")("Slider")

            }

            else if (nrOfSubs == 5) {

              thisComp.layer("subselect5").effect("Slider Control")("Slider")

            }

            else if (nrOfSubs == 6) {

              thisComp.layer("subselect6").effect("Slider Control")("Slider")

            }

            else if (nrOfSubs == 7) {

              thisComp.layer("subselect7").effect("Slider Control")("Slider")

            }

            else if (nrOfSubs == 8) {

              thisComp.layer("subselect8").effect("Slider Control")("Slider")

            }

            else if (nrOfSubs >= 9) {

              thisComp.layer("subselect9").effect("Slider Control")("Slider")

            }

            else {

              thisComp.layer("subselect3").effect("Slider Control")("Slider")

            }

             

             

             

             

            Also the same for its Opacity:

            nrOfSubs = parseInt(thisComp.layer("_Info").text.sourceText.split("\r")[2])

            if (nrOfSubs <= 3) {

              thisComp.layer("subselect3").effect("Slider Control").compositingOption.effectOpacity

            }

            else if (nrOfSubs == 4) {

              thisComp.layer("subselect4").effect("Slider Control").compositingOption.effectOpacity

            }

            else if (nrOfSubs == 5) {

              thisComp.layer("subselect5").effect("Slider Control").compositingOption.effectOpacity

            }

            else if (nrOfSubs == 6) {

              thisComp.layer("subselect6").effect("Slider Control").compositingOption.effectOpacity

            }

            else if (nrOfSubs == 7) {

              thisComp.layer("subselect7").effect("Slider Control").compositingOption.effectOpacity

            }

            else if (nrOfSubs == 8) {

              thisComp.layer("subselect8").effect("Slider Control").compositingOption.effectOpacity

            }

            else if (nrOfSubs >= 9) {

              thisComp.layer("subselect9").effect("Slider Control").compositingOption.effectOpacity

            }

            else  {

              thisComp.layer("subselect3").effect("Slider Control").compositingOption.effectOpacity

            }

             

             

             

             

            There is also 1 box as main catagory, changed up to 3 times. It uses identical code in 3-Nulls, that are read by a main-Null.

             

             

            The colors used for the boxes is determined by a composition with the general settings. The boxes use the fill-color with:

            comp("_GeneralSettings").layer("Colors").effect("Color_Sub")("Color")

             

             

            That general color gets its color through:

            evalSeries = thisComp.layer("Series").transform.opacity;

            if (evalSeries==1) { [240,125,0,255]/255 }

            else if (evalSeries==2) {[0,79,159,255]/255}

            else if (evalSeries==3) { [227,6,19,255]/255 }

            else {[0,0,0,1]}

             

             

             

             

            As mentioned, there is a text-field called _Info, which holds the combined data to be shown.

             

             

            For the 9 lines in the main-catagory box, there is car-nuber, car-class, team name, car make + type, time/lap-difference and the country-flag

            Each of these fields is again controled by a null. It does set the Position:

            guideLayer = thisComp.layer("adjust-LocalVars").effect("adjustFlag")("Layer");

            thisNumber = thisLayer.name.split("|")[1];

            timeAdjust = (thisNumber-1) * comp("_GeneralSettings").layer("vars-Standings").effect("flag-in-frames")("Slider")*(1/25 );

            ownStart = transform.position.valueAtTime(0)

            guideStart = guideLayer.transform.position.valueAtTime(0)

            if (time > (5/25)) { ownStart + (guideLayer.transform.position.valueAtTime(time-timeAdjust) - guideStart) } else {value }

             

             

            And opacity:

            guideLayer = thisComp.layer("adjust-LocalVars").effect("adjustFlag")("Layer");

            thisNumber = thisLayer.name.split("|")[1];

            timeAdjust = (thisNumber-1) * comp("_GeneralSettings").layer("vars-Standings").effect("flag-in-frames")("Slider")*(1/25 )

            if ((guideLayer.transform.position.key(1).time+timeAdjust) > time)

            { 0 } else { parent.transform.opacity}

             

             

             

             

            The above example is for the country-flag, below for the number:

            Position:

            guideLayer = thisComp.layer("adjust-LocalVars").effect("adjustNumber")("Layer");

            thisNumber = thisLayer.name.split("|")[1];

            timeAdjust = (thisNumber-1) * comp("_GeneralSettings").layer("vars-Standings").effect("number-in-frames")("Slider")*(1/ 25);

            ownStart = transform.position.valueAtTime(0)

            guideStart = guideLayer.transform.position.valueAtTime(0)

            if (time > (5/25)) { ownStart + (guideLayer.transform.position.valueAtTime(time-timeAdjust) - guideStart) } else {value }

             

             

            Opacity:

            guideLayer = thisComp.layer("adjust-LocalVars").effect("adjustNumber")("Layer");

            thisNumber = thisLayer.name.split("|")[1];

            timeAdjust = (thisNumber-1) * comp("_GeneralSettings").layer("vars-Standings").effect("number-in-frames")("Slider")*(1/ 25);

            ownStart = transform.position.valueAtTime(0)

            guideStart = guideLayer.transform.position.valueAtTime(0)

            if (time > (5/25)) { ownStart + (guideLayer.transform.position.valueAtTime(time-timeAdjust) - guideStart) } else {value }

             

             

             

             

            The Fields are populated by the parent of the above null

            The source text for the car number is:

            try {

            var thisLine = parent.parent.text.sourceText

            thisLine.split("~")[0]

            } catch(err) { }

             

             

            That parent is a hidden text field, that gets his data like this:

            Source text:

            valSlider = parseInt(thisComp.layer("mainselect").effect("Slider Control")("Slider"))+1;

            try {

            thisComp.layer("_Info").text.sourceText.split("\r")[valSlider]

            } catch (err) {}

             

             

            And opacity:

            var thisLine = text.sourceText;

            if (parseInt(thisLine.split("~")[0])>0)

            {

            (parent.transform.opacity/100) * (thisComp.layer("mainselect").effect("Slider Control").compositingOption.effectOpacity/100) * 100

            } else {0}

             

             

            The referenced MainSelect, is the above mentioned null, that collects its data from the null's, just like the sub-selects.

             

             

            As mentioned, the problem might lie in the flags. There are 9 flags (only for the main catagory box). Each flag has a time remap, set like this:

            n = Math.floor(effect("Slider Control")("Slider"));

            framesToTime(n);

             

             

            The slider gets the data from the country name, as the corresponding composition has 1-frame flags for each country. 1 per layer:

            var thisLine = parent.parent.text.sourceText

            try {

              flag = "flag_" + thisLine.split("~")[8]

              comp("CountryFlags").layer(flag).index

            }

            catch(err){

              0;

            }

             

             

             

             

             

             

            The sub catagories have no flags, and only number, team and car text-fields. They are filled identically to the main catagory.

             

             

            In addition to this, there is a call to 2 compositions, which together show the series logo and circuit data. These are shown through a scroll to combined images of all events and stops on the correct one. No time remapping.

             

             

             

             

             

             

            It is quite a bit, but then again, a lot of data is to be shown, and in an appealing way

             

             

            As mentioned, the standings composition sports 326 layers, and does call some other compositions too. But that shouldn't result in a 2 hour rendering time, for a 45 second animation... (I would think)

             

             

            Any help is greatly appreciated.

            • 3. Re: 45 second AE with expressions, takes 2 hours to render
              Mylenium Most Valuable Participant

              But that shouldn't result in a 2 hour rendering time, for a 45 second animation...

               

              Sure, why not? As I already said, your stuff is chocka full with temporal calculations such as nearestKey(), framesToTime() and valueAtTime(). What happens when you use them is that AE combs through every layer and pre-comp and retrieves the calculated time() value rather than the native time value and the property state at that time. Also branching out in conditional statements won't improve matters. Inevitably, in a 326 layer comp that is bound to cause some massive overhead. I can only repeat what I said already: You need to restructure your project. It really seems to me that you've gone trigger happy on rigging everything with expressions when rather you could simply turn off layer visibilities and group items in more pre-comps, which is another ugly truth here: setting opacity to zero will still force AE to evaluate expressions, even if a layer is invisible. In fact you may even introduce more unnecessary processing when your calculated values for the fades and whathaveyou en up being infinitessimal fractions - you already can't see them, but your layers are still there. Again, this is nothing you can fix with just rewriting the expressions. Your whole approach is somehow crooked. Sorry, but I can't really put it another way. You have "over-efficient" yourself to death here and AE's expression engine just isn't up to it.

               

              Mylenium

              • 4. Re: 45 second AE with expressions, takes 2 hours to render
                marque-1968

                First, before I forget, thank you for spending this time reading and answering my question. It is really appreciated.

                 

                You mention I must use Native Time Values. That sounds like a smart move. However, later you state that rewriting expressions won't help.

                 

                Even though I programmed for decades already, programming for AE is rather new to me, so any help is appreciated.

                 

                Last year I made the same show, and used active scripting to create the compositions out of a template, changing flags and texts in code. The graphical background was in a pre-rendered secondary comp. This pre rendering mainly to speed up Premiere Pro, as I used dynamic linking to AE at the time.

                 

                How I read your response, is to go back to that original idea. Leave expressions to a bare minimum and return to scripting. Or is it really down to my own abilities and is it easily possible to get all this variable data (including variable in amount) shown in an identical way using mainly expressions?

                • 5. Re: 45 second AE with expressions, takes 2 hours to render
                  Rick Gerard Adobe Community Professional & MVP

                  When you start writing expressions based on time it's very easy to end up with a recursive calculation that has to start back at frame 1 and analyze every other frame. This happens because AE's expression language does not remember or store the data calculated from the last frame. Many of your expressions fall right into this trap.

                   

                  There is some nicely written code and some creative thinking that has gone into your expressions but from what I see the project could have been simplified quite a bit by just inserting a couple of keyframes on a couple of layers here and there that would have removed the need to keep looking for the data from the nearest keyframe.

                   

                  As far as pre-rendering for speeding things up, that all depends on the project. If I rotobrush for more than a few seconds I always render a lossless with alpha DI and replace the RotoBrush layer. I then completely get rid of the RotoBrush comp. There are a bunch of other situations where I'll render and replace usage in AE to speed things up. I never use Dynamic link for complex composites and complex motion graphics because the rendering time through Dynamic Link in Premiere Pro can easily be 4 or 5 times longer than the rendering time from AE. I do copy and paste footage or even use Dynamic Link to generate an AE project for a complex AE comp, but I then go right back into Premiere Pro and undo the Create Comp from layer. After the effect has been rendered I just replace the footage in the PPro timeline.

                   

                  I hope these suggestions help. It will take a while to figure out when you have a recursive expression and when it will kill your render speed. It will also take some time to know when you should render and replace.

                  • 6. Re: 45 second AE with expressions, takes 2 hours to render
                    Mylenium Most Valuable Participant

                    The issue is not per se with your code, it's simply AE being inefficient at evaluating expressions and you are using stuff that basically tells AE "Does this element exist at time x? Which value has it then?" which under the hood makes AE run a million evaluation cycles back and forth plus of course may trigger secondary phenomena like AE actually spending time to decode footage for that remote point in time to evaluate the outcome. With regards to the expressions this simply means that your expressions must be aimed at avoiding such stuff wherever possible. By default AE's global time (= "synchronize time of related items" in the preferences) takes care of keeping things temporally aligned and this is basically what you should use. It would be much more efficient to create longer comps and just let time run continuously rather than force AE to freeze frames/ calculate static values via expressions. A simple optimization would e.g. be to not have your flags staggered in time, but rather simply have them stacked inside a pre-comp and then only apply straightforward on/off expressions to opacity where your actual animation is plain keyframes in the parent comp as opposed to calculating the fade on some value. Of course you won't be able to avoid some of these things, but I really think you simply need to work more with AE's native functions, not against them plus also develop an understanding for how the program caches states and pixel buffers in its processing chain, meaninghow. e.g. pre-comps may only be evaluated once if nothing changes in them over time.

                     

                    Mylenium

                    • 7. Re: 45 second AE with expressions, takes 2 hours to render
                      marque-1968 Level 1

                      Thank you Mylenium and Rick Gerhard,

                       

                      Due to your suggestions, I have gotten back into programming it all via scripting, in stead of expressions. Flags for instance are now fixed, and the layers are split for each flag.

                       

                      I had also removed all the graphics, boxes and texts that are default for the series, so they could be pre-rendered.

                       

                      I was now contemplating to changed all the graphic boxes (created in Photoshop) to After Effects solids with masks. Perhaps not having to use graphics, could speed up everything even more. So I decided to test the compilation duration.

                       

                      I used a standings and measured the time until all the flags were drawn for the first time, and stopped moving.

                       

                      I tested without the background animations and with flags being fixed images.

                       

                      - Using graphics: Duration was about 5 minutes, 23 seconds.
                      - Using AE Solids: Duration was aboutn 5 minutes, 20 seconds.

                       

                      3 seconds difference, and as it was done manually, the difference could also be explained due to the time it took me to decide that the movement had stopped.

                       

                      But then I had a revelation, what if I remove motion blur.
                      - Using AE Solids, and no motion blur: 1 minute and 20 seconds.

                       


                      So that triggered me to go back to the original issue:

                       

                      I now used an unaltered composition for the standings
                      - with motion blur: 8 minutes and 40 seconds
                      - after removing the motion blur: 1 minute and 35 seconds.

                       

                      So the real benefit is in removing the motion blur. I find the motion blur to be more appealing, as it looks more like a real movement and was aware that this would cause some extra overhead. However, I was not aware that this overhead was nearly a delay factor 6.

                       

                      I have now run the original script whole again
                      - With motion blur: 1h 45 min
                      - Without motion blur: 16 min.

                       

                      As mentioned before, I am happy to have the versitily of having one "_INFO" layer, and each change in that, or in the general settings of the project, is reflected imediately.

                       

                      When using a composition with fixed graphics and a precompiled background, I can save 15 seconds or 13%. I think it worth it for me to take that overhead, and lose the motion blur overhead. It is mainly something I will notice and some TV-Makers, but the average viewer won't notice at all. Whilst when a new standing for the show needs to be created just before the deadline (as happened) I don't need about 13 hours, but less then 2. That is something I can overcome.

                       


                      Thanks again for your assistance, and forgive me for not thinking about the motion blur before...

                       

                       

                       

                       

                      Marque.