10 Replies Latest reply on Feb 9, 2012 1:45 AM by Mike Basil

    Multiprocessor Check-box for Renders

    Toomany3 Level 1

      A very quick rant and then a question:

       

      Why in the world do I have to figure out what way is the best to render my projects.  Check the box or not?  If so, then what's the right balance between cores and ram?  Not a big deal for 15 minute renders... one way could be 14 minutes, the other 16.  No big loss.

       

      But on those big 30 hour renders, the wrong setting could mean 30 hours or 45 hours.  I just did one today that was 3 hours one way and 15 hours the other.

       

      What I do right now is render out a small portion each way, giving it enough time to give me a good ETA.  (I'm starting to get a feel when the check-mark works and when it hurts). 

       

      But if I could do it, being a stupid human with 100 other things to do and think about, why can't my computer (ahem, After Effects) who has NOTHING TO DO ALL DAY but ONE SINGLE THING (you know what I mean, it doesn't have to pick up kids from school, pay taxes, sleep, buy groceries, etc.), why can't AE pick the best settings for me and help me take a small part of my life back?

       

      Surely if I can do it by trial and error, AE should be able to do it with a little logarithm by seeing what effects are turned on, etc. and say, "gee, this is a job for multiprocessor rendering... I'll check the box for him so he can stuff his face with a Taco Bell burrito."

       

      But, no!  I can't eat Taco Bell because I'm trying out 3 different render settings and trying to find that proper balance. 

       

      Rant's over.  Sorry; it went longer than intended.

       

      Now, is there a better way to do the above without all the trial and error?

        • 1. Re: Multiprocessor Check-box for Renders
          Toomany3 Level 1

          Here's a perfect example.  I've been testing out a render for a sort of template we will be using on future projects.  These projects end up being about 6 minutes of final video total.

           

          A 30-second greenscreen key with some lights/shadows, cam, 3D set, etc.  I'm rendering off of an overclocked i7 980X 6-core (4.3ghz) with 24GB ram.  Here are the different ways I've rendered it and the vastly different results:

           

          MP = Multiprocessor option.  The first numbers represent my render time in MM:SS.

           

          34:40 - No MP

          21:48 - MP, 0 cores for other apps, 3GB per core, 2 cores rendering

          17:45 - MP, 0 cores for other apps, 2GB per core, 4 cores rendering

          31:30 - MP, 0 cores for other apps, 1.5GB per core, 5 cores rendering

          37:11 - MP, 0 cores for other apps, 1GB per core, 9 cores rendering

           

          You can see how finding that sweet spot can really make a difference when you're rendering out a longer file.  The two extremes are just about 1:2. 

          • 2. Re: Multiprocessor Check-box for Renders
            Mylenium Most Valuable Participant

            Now, is there a better way to do the above without all the trial and error?

             

            No. That is, not until the day AE gets a new render pipeline with better adaptive threading, microbucketting and all the usual cleverness plus better temporal handling. In such a world the app would simply continually probe for available resources and fire threads and sub-threads with minimal memory footprints rather than using separate processes. And such a processing model could of course use CUDA and otehr hardware acceleration or even distributed rendering in a network/ the cloud. But alas, we're not there yet and given, how Adobe has procrastinated on this for eons and instead sold us the failed promise of OpenGL, we may have to wait another few versions...

             

            Mylenium

            • 3. Re: Multiprocessor Check-box for Renders
              Toomany3 Level 1

              You know, another thing that's really dumb about this whole thing is that I've noticed it doesn't really matter WHAT settings you give it.  If you find the sweet spot settings, AE ignores the settings you give it anyway.

               

              For example.  I'm running a render right now.  I'm on my render machine that's 32 real cores (dual AMD 6727).  It couldn't use all 32 cores at 3 GB ram/core.  It did its calc thing and said that using 32 cores would require using only 1.5 GB ram.

               

              I know that for this render 1.5GB/core isn't enough ram, so I bumped it up to 2GB/core.  It said it could only use 24 cores.  Ok, better than nothing.

               

              So i fire up the render and after a few minutes I runm the task manager to see how much of the system it's using, and what do you know?!  I found the sweet spot and--despite it telling me I was only going to be using 24 cores--I'm using all 32 cores at a full 100%.

               

              Those are numbers you always like to see--100%.  But I only get 100% activity on ALL cores on this rig when I tell it NOT to use all its cores for rendering.

               

              And when I tell it TO use all its cores (albeit at a lower ram/core ratio), it only used all 32 cores at 40-60%

               

              I think the trick is to find that sweet spot.  It varies from compute to computer, project to project.

               

              Capture.PNG

              • 4. Re: Multiprocessor Check-box for Renders
                Mylenium Most Valuable Participant

                Well, there are internal safeguards and metrics at work that try to be all too helpful. It's AE's eternal failing - in trying to be too smart, it actualyl looks really, really dumb and prevents you from getting things done. Well, whatever. We just have to wait and see.

                 

                Mylenium

                • 5. Re: Multiprocessor Check-box for Renders
                  Todd_Kopriva Level 8

                  You're misunderstanding the meaning of the settings.

                   

                  When you tell After Effects how many CPUs to use, you're specifically telling it an upper limit of how many background processes to start (one per CPU) when you're using Render Multiple Frames Simultaneously multiprocessing. This setting, however, does not limit the number of processors that can participate in multithreading. Multithreading is independent of Render Multiple Frames Simultaneously multiprocessing.

                   

                  This video demonstrates the difference:

                  http://www.video2brain.com/en/videos-5355.htm

                  • 6. Re: Multiprocessor Check-box for Renders
                    Toomany3 Level 1

                    Thanks, Todd.  That clears up the misunderstanding I had.  I see that using cores and multiple frames at the same time are two different things.  I was lumping the two concepts together. That link was helpful.

                     

                    However, I guess that the semantics are out of the way, why should we have to render out our projects with different settings (box on/off, and fine tuning the ram/core ratio) in order to know which settings happen to be best for this particular project?

                     

                    It wouldn't be so bad if you only had to set it once and forget about it.  But every project is different and unfortunately, you one setting that might rock in one project could be the absolute worst in another.

                     

                    I posted some real results of the same render above, but with different settings. If you get the sweet-spot settings your render is literally TWICE as fast as if you pick the wrong settings.

                     

                    I just did another longer test last night.

                     

                    4:40 hrs. one way, and I'm going on 12 hours now with about 10% to go.

                     

                    EXACT SAME PROJECT

                    EXACT SAME EFFECTS

                    EXACT SAME EXPORT SETTINGS

                     

                    The ONLY thing that's different is my ram-to-core ratio.

                     

                    I think that's our biggest gripe. The only way I know how to find the sweet spot is really trial-and-error.

                     

                    If I can do it, why can't AE?

                     

                    Why can't AE cycle through the 4 (or so) settings as it renders automatically and say:

                     

                    Option 1: 10 frames per minute

                    Option 2: 5 frames per minute

                    Option 3: 15 frames per minute

                    Option 4: 13 frames per minute.

                     

                    DING, DING! We'll go with option 3.  Looks like it's the best for this render.

                     

                    I know I'm not a developer and I'm sure it's a bit more complicated than that, but it seems like this is exactly what we (users) do when we render, so why couldn't AE do the same until a better solution is made?

                     

                    $0.02.

                    • 7. Re: Multiprocessor Check-box for Renders
                      Todd_Kopriva Level 8

                      We also want to have After Effects take care of this for you, but it is not a trivial problem to solve. We are working on it, though.

                      1 person found this helpful
                      • 8. Re: Multiprocessor Check-box for Renders
                        TheFosterHouse Level 1

                        I came here looking for a solution to this as well. I've found that this issue isn't just that you have to find the sweet spot for your machine, but you also have to find the sweet spot for the files that you're using. This is mind numbing due to Multiprocessing needing to be changed on a per project basis. I use to do that trail and error thing, but it was just such a time sink I stopped seeing a time saving benefit in Multiprocessing.  I'm running a 12 core Mac with 64Gb of Ram with source stored on a Dual SAS raid 10 and AE still chokes half the time if Multiprocessing is on. This usually results in the entire system becoming unresponsive and render times taking many times the amount of a regular render. If I'm lucky, I can stop the render before it gets completely unmanagable. I've been trying to nail down what differences in projects cause this while trying to adapt the settings in AE, but it's just maddening.

                        • 9. Re: Multiprocessor Check-box for Renders
                          Toomany3 Level 1

                          TFH, I know the feeling well.... "Is it going to crash?!?" and holding onto hope.  I've just gotten into the very strict habit of ALWAYS saving before a render (I know, I should anyway), but I set my render settings and then save again.  Because 20% of the time the computer will not even render and I know I'll have to force close.

                           

                          Oh well, hopefully CS6 brings us better tales.

                          • 10. Re: Multiprocessor Check-box for Renders
                            Mike Basil Level 2

                            I must agree with Todd, even with multi-processing off, multi-threading can and will (depending on the case) still be used when rendering a single frame.

                             

                            Also to mention AE is not just an application, but the application and all the plugins that are available to it (including PPro plugins).

                             

                            Imagine a plugin where an effect would use the result of the previous frame to compute the next, what do you do?!...The hardest part is to make developers to adapt to the changes and I recon Adobe team knows about it.