24 Replies Latest reply: Jun 20, 2009 10:21 PM by bsnyder175 RSS

    AE rendering error (memory leak?)

    bsnyder175 Community Member

      I bought a new computer about a year ago, which I believed to be more than enough to power AE and the other apps I use with it.  It's an 8-core 3.2 GHz Mac with 4 gigs of ram.  However, AE is constantly erroring out during a render test of mine.  Nothing more than basic screen screen (subjects are blacked out) with a jpg background.  It's about 35 minutes long.  It error'd probably 3 or 4 times before I lowered the composition size from 640x480 to 320x240.  One render made it through successfully in this size.  I made an extremely slight change (just moving one of the subjects) and did another test render of 2 minutes.  It was flyin right through and again, the render stopped saying it "Failed."  It never produces error log files, so I have no idea what's going on.

      After a failed attempt, I'll try it again and the program stops responding when I hit Render.  From there, I have to force-quit and try again.

      I've turned on my Activity Monitor to see how much memory it's soaking up, and it's never anymore than ~2 gigs.  Is this a memory issue?  I find it completely hard to believe with this system.

      I would REALLY appreciate any knowledge that anyone can give me regarding the situation.

      Edit:  the last render I did was again a 2 minute-length video.  2 minutes, 18 seconds into the render process, it failed.  Activity Monitor showed it stopped at 2.84 gigs of memory (a constant climbing number until it failed there).  It also displays "AE (null)" as the title of the process.  And, after a failed render, AE sits at 2.84 gigs of memory.  That memory never gets freed back up, which I'm guessing is why the application becomes non-responsive.

      Edit 2: After looking at how much Free memory I have, it shows around ~2.5 gigs (that's with only AE and firefox open).  If this is a memory problem, how much do I need to execute simple renders like this in AE?  All it is, is a 320x240 key'd-out, silhouette subject with a different background (ideally an hour long - but even 2 minute-length fails).  Should AE really need to use ~3 gigs of memory for this task?  I am more than willing to put a few more gigs of ram into this computer if necessary, but I don't want to add more if AE will use all of that ram up also, to the point of crashing.  I'm wondering if it's a problem beyond system requirements (which this computer should meet 3 times over).  It's really frustrating.

        • 1. Re: AE rendering error (memory leak?)
          Adolfo Rozenfeld Adobe Employee

          4 GB of RAM is only enough to feed 2 cores, when using After Effects multiprocessing (Render Multiple Frames Simultaneously).

          And definitely inadequate for an octo-core machine.

          If you enable multiprocessing for all 8 cores with such configuration, the backround rendering instances will starve to death, or keep shutting and re-launching.

          While you only have 4 GB, I suggest you go to Memory and Multiprocessing preferences and set "Leave CPUs freee" to 6 cores (ie, only use 2).

           

          If you really want to use all 8 cores for rendering, it's recommended you get 2 GB per core (ie, 16 GB total). After all, you're running a virtual render farm of sorts inside your computer. You can get away with about 1.5 GB per core (12 GB total) if you're not doing HD video and/or use using high bit depths (16 or 32 bpc) in your projects.

           

          But it's not an all or nothing kind of thing- You could get an extra 4 GB (8 GB total, by all means you want at least that amount) and use 4 cores for rendering in AE (ie, leve 4 CPUs free). That would give you very, very good performance.

           

           

          I've turned on my Activity Monitor to see how much memory it's soaking up, and it's never anymore than ~2 gigs

           

           

          That's for the main After Effects instance. If you're using multiprocessing, each rendering instance would appear as "AEselfLink".

          • 2. Re: AE rendering error (memory leak?)
            bsnyder175 Community Member

            Thank you so much for the response.  I put in an order from Crucial for 4 more gigs (total of 8).

             

            When I looked for the preference though, in CS3, to set the number of cores to use, I wasn't able to find it.  Under Memory & Cache, it shows Max Memory Usage [120] % = 3.0 Gigs and Maximum RAM Cache Size [60] % = 1.8 Gigs.

             

            Under Multiprocessing, the only option there was a checkmark for Render Multiple Frames Simultaniously.  It was already checked off.  Turning this option on didn't open up any further options for specificying how many cores to use.

            • 3. Re: AE rendering error (memory leak?)
              Adolfo Rozenfeld Adobe Employee

              The preference for setting the number of cores is new in CS4.

              In CS3, there could be a way to set this in the text preferences, but it's not going to do any good if multiprocessing is off.

              Everything I wrote only applies if Render Multiple Frames Simultaneously is on (which is the main point of using a multi-core machine for AE!)

               

              If you're using a small composition size with small source files, there's nothing wrong with AE not using memory it may not need. Some Compositions are memory intensive by nature, and others aren't.

               

              How are you exporting? Are you using the Render Queue or File > Export? What format/codec are you rendering to?

              • 4. Re: AE rendering error (memory leak?)
                bsnyder175 Community Member

                I'm using a 6.6 gig source file (which I could probably compress).  In the composition, I'm using the same source file twice, in two layers.  To export, I'm going to Make Movie... which adds it to the render queue.  The codec is Animation (Quicktime).

                 

                Would using a much smaller source file help?  The soure file was generated simply by using iMovie to import from a miniDV.  There wasn't any compression added after the raw import.

                 

                Is there any way you can direct me to the text file I can edit to be able to free up x amount of cores?

                • 5. Re: AE rendering error (memory leak?)
                  Adolfo Rozenfeld Adobe Employee

                  No, you shouldn't have to compress your source footage, unless we're talking extraordinarily large files (unlikely).

                  Nope, there are no cores to free because you are not using that feature at all. This setting only is used when you have Multiple Render Frames Simultaneously enabled.

                  With that feature turned off, 4 GB of RAM shouldn't cause problems (of course, taking advantage of your hardware would require enabling it and getting more RAM).

                   

                  Is it possible that the target drive doesn't have enough room for the exported file?

                   

                  Did you update After Effects CS3 to the latest update available for it (8.02)? If you didn't, just go to Help > Updates and the Adobe Updater should offer that update.

                  • 6. Re: AE rendering error (memory leak?)
                    bsnyder175 Community Member

                    I was saying I would turn the option on to use more cores when the extra 4 gigs of ram comes in - forcing me to edit this text file to use only 4 cores.

                     

                    I have version 8.0.2.27.  My target drive has 132 gigs of free space to use.  Maybe a reinstall?

                    • 7. Re: AE rendering error (memory leak?)
                      Adolfo Rozenfeld Adobe Employee

                      This is for CS3 only.

                       

                      The text prefs file is in UserName/Library/Preferences/Adobe/After Effects/8.0/Adobe After Effects 8.0 Prefs

                       

                      Make sure you quit the application. Open the text preference file and look in the ["MP"] section for: “MaxNumberOfProcesses” = “0″.

                      “0″ is the default setting, which disables this preference setting. For better performance on an 8-core machine with 8GB RAM, for example, change the “0″ setting to “4″ and save the preference file and restart After Effects. This restricts multiprocessing with Render Multiple Frames Simultaneously to 4 cores, each getting approximately 2GB of RAM.

                       

                      You could right now set it to "2", and AE will use just two cores to render multiple frames simultaneously.

                       

                      BUT, this is unlikely to be related to your problem, since this feature was disabled completely.

                      • 8. Re: AE rendering error (memory leak?)
                        bsnyder175 Community Member

                        I'll try it when the extra ram gets here, but if extra ram doesn't solve the issue, I'm at a loss.  I don't know what to do.  The majority of the renders I've done with multiprocessing off, the AE memory usage has been 1.83 or 1.85 gigs (nearly every time).  Sometimes this raises until I'm out of memory.  The last render I did about 40 minutes ago failed when I still had over a gig of free memory at the time of failure.  Do you think reinstalling AE would do anything?

                        • 9. Re: AE rendering error (memory leak?)
                          bsnyder175 Community Member

                          After another test, I'm convineced there is some kind of memory leak.  About 17 minutes into the render process, the memory usage never went above or below 1.83 gigs.  Then all of a sudden around 20 minutes in, it went to 2.84 gigs and failed (also becoming non responsive).

                           

                          Is this memory behavior normal even under regular circumstances, having multiprocessing turned off?

                          • 10. Re: AE rendering error (memory leak?)
                            Adolfo Rozenfeld Adobe Employee

                            There are some projects which can fail to render due to excessive memory usage, but usually these make use of source files with very, very large dimensions in pixels.

                            To get you out of trouble in this case, you can simply render the first half of the Comp (Setting time span to Custom in the Render Settings and enter the duration manually) and then the other half. If you have Quicktime Pro, you can then open both files in QT Player, select all (Command+A) and copy the second file, paste it at the end of the first and save. That's it.

                            Also, try downloading the CS4 trial and see if it manages to get to the end of the render?

                            • 11. Re: AE rendering error (memory leak?)
                              bsnyder175 Community Member

                              I actually tried that in one of the earlier tests.  I went to render settings and rendered only the first 2 minutes of the composition - still failed.  I'll download the cs4 trial and see what happens.  Thanks a lot for sticking around with this issue.  I really do appreciate your assistance in solving this.

                              • 12. Re: AE rendering error (memory leak?)
                                TimeRemapper Community Member

                                Have you tried disabling layer caching and purging during your render?

                                 

                                If you hold down the "shift" key when you access the AE preferences, and then release the key when the preferences dialog box pops up, you'll be able to access a hidden item in the pulldown menu entitled "secret". When you go there, you can choose to purge every "1" frame during rendering, and you can disable the layer cache.

                                If this works, don't forget to go back to the preferences after the render completes and restore the settings to their default states.

                                 

                                Aside from that, is there anything occuring in your project's timeline around the area your crash/hang is happening?

                                Any effects being applied? Any footage element beginning?

                                • 13. Re: AE rendering error (memory leak?)
                                  bsnyder175 Community Member

                                  Steve,

                                   

                                  I received this solution from another person also, which is a test that's currently rendering right now.  I'm having it purge every 100 frames.  I didn't try disabling the layer cache - I'll try that also if this doesn't work.  I'm about 10 minutes into this render and it's taking about half of what is usually takes (with the purge idea).  Average of 500 megs of memory usage from AE, vs the usual 1.85 gigs.

                                   

                                  The composition is extremely simple, imo.  This is what it is:

                                   

                                  background layer is a library (with 2 chairs) cg scene, which is just a jpg.  I filmed two men sitting down having a converstaion in front of a green screen, which I keyed out (just the 2 men) and then applied a hue/sat modifier to decrase the brightness to -100.  That's it.  Comp size is 320x240.  I keyed them seperately from the same source file (2 layers) and positioned them in the chairs of the background scene.  I am *completely* baffled that something which seems so simple, especially given the comp size, is requiring so much memory to render.  But then again, I'm no video editing guru either.  Maybe this is normal - I don't know.

                                   

                                  One question I do have.  Does comp length have anything to do with memory usage?  Would a 5 minute comp take just as much memory as a 6 minute comp?  Or does length make the memory usage vary?

                                  • 14. Re: AE rendering error (memory leak?)
                                    Todd_Kopriva Adobe Employee
                                    > Does comp length have anything to do with memory usage?  Would a 5 minute comp take just as much memory as a 6 minute comp?  Or does length make the memory usage vary?

                                     

                                    When you're rendering for final output, you only need to consider the amount of RAM needed to render each frame. If you're using time-related effects or motion blur or other temporal things, then some surrounding frames must also be considered. But the whole rendered composition isn't being retained in RAM or anything. (When you're rendering a RAM preview, you're also using the RAM allocated to the foreground process to store the frames for playback.)

                                     

                                    So, basically, the memory usage to render a composition for final output doesn't change with the length of the composition.

                                     

                                    I'm glad that purging memory periodically is helping you to get your composition rendering.

                                     

                                    It seems that you're past this issue right now, but I wanted to say this for others reading this thread:

                                    To echo something that Adolfo said: Don't enable Render Mutliple Frames Simultaneously if you're failing to render without it. Only enable Render Mutliple Frames Simultaneously if you're rendering just fine without it and want to see if you can get a speedup by throwing another processor core at the render. The only problem that Render Mutliple Frames Simultaneously ever solves is the problem of slowness---and it only solves that problem some of the time... and it can only do that when you have lots of RAM. This preference defaults to OFF for a very good reason.

                                    • 15. Re: AE rendering error (memory leak?)
                                      bsnyder175 Community Member

                                      Tried it with cs4 (MP turned off).  It didn't have keylight installed so I just took a copy of the keylight plugin from cs3 and threw it into cs4 (came up with an invalid lisence but still worked with a watermark).  It failed again about 5 minutes in - same results.  At the time of the failure, AE had taken up 2.6 gigs of ram, leaving me with next to nothing.  At this point, can we safely assume that it's not AE, and it's my system?  I'm really hoping the extra 4 gigs will solve my issue.  This comp is something I'll be putting together on a near weekly basis, so I'd rather have more memory fix the issue than relying on frame-purging, if possible.

                                      • 16. Re: AE rendering error (memory leak?)
                                        Mylenium MVP

                                        bsnyder175 wrote:

                                         

                                        At this point, can we safely assume that it's not AE, and it's my system?

                                         

                                        It's neither, I daresay. It's probably down to Keylight. It's a memory hog that can slow down even beefy systems. Also there are some bugs in it, especially when using the CS3 version in CS4, which may have contributed to the failure of your test. The additional RAM will certainly improve things. However, I'd definitely also consider trying a different keyer like Primatte at some point. Overall behavior may improve considerably just by that.

                                         

                                        Mylenium

                                        • 17. Re: AE rendering error (memory leak?)
                                          bsnyder175 Community Member

                                          The memory came in today. Tried it out with one keylight in the scene (one not rendered). Had 4 cores going with 8 gigs of memory. It rendered just fine. Had around 2.5 gigs of memory free by the time was render was done.

                                          Tried it with both people in the scene (2 layers using keylight). Ran outa memory. I wasn't paying attention to the activity monitor. Tried another render, this time with 2 cores on. Ran out of memory again; however, Activity Monitor said it had 3 gigs of memory free to use at the time of the error message.

                                          Now, I'm actually receiving error messages (pop ups) that say:

                                          After Effects error: Keylight out of memory. (4)
                                          (25 :: 241)


                                          In all of my earlier tests with 4 gigs of memory, I wasn't getting this error. It would just quit and the render status said Failed.

                                          I took a screenshot of the results below.  I take it something more is wrong if AE is telling me I'm out of memory when I clearly have ~3 gigs left.

                                           

                                          On purchasing Primatte, I don't have the funds to put down $500 just to key out video.  I'd rather use Keylight if possible.

                                           

                                          Picture 1.png

                                          • 18. Re: AE rendering error (memory leak?)
                                            Adolfo Rozenfeld Adobe Employee

                                            bsnyder:

                                            It seems Keylight is devouring your RAM over time.

                                            This looks like a bug.

                                            I have done hundreds of shots with the version of Keylight that comes with CS4, and at much larger frame sizes.

                                            So, I'd like to take a look at your project.

                                            How many GBs would it be for the folder containing the .aep project file + source files? About 7 GBs, I think you said? If you zip that folder, does it come down in size? I am asking to see if you want to upload that folder so I can take a look and see if there's something

                                             

                                            As always, I have to make it clear that I can't promise an immediate solution, but rather investigate the issue to isolate the causes, notify the developer and have it fixed in the future. Of course, if I do find a woraround I'll let you know.

                                            • 19. Re: AE rendering error (memory leak?)
                                              bsnyder175 Community Member

                                              Do you have a server I can upload it to?  Can I also have your email address?  I'd like to send you some private information.

                                               

                                              Thanks!!

                                              • 20. Re: AE rendering error (memory leak?)
                                                Adolfo Rozenfeld Adobe Employee

                                                My e-mail address is arozenfe[[AT]]adobe.com

                                                 

                                                Let me investigate the server thing. I'll send a private message once I know, ok?

                                                • 21. Re: AE rendering error (memory leak?)
                                                  bsnyder175 Community Member

                                                  sounds great.

                                                  • 22. Re: AE rendering error (memory leak?)
                                                    bsnyder175 Community Member

                                                    I'm wonder if this may have solved it...

                                                     

                                                    I turned the max memory usage down from 120% to 60%.  Can someone explain why the default would be 120%?  Wouldn't that make *everyone's* computer crash by default?  If I understand correctly, that means it will allow AE to use up every bit of memory that the computer has to spare, allowing memory-draining crashes.

                                                     

                                                    What is the difference between the max memory and max cache memory?  Because lowing the max memory (120% default value) seemed to fix the issue.

                                                     

                                                    - update:  nevermind   Render went about halfway through until it gave me the image buffer error.

                                                    • 23. Re: AE rendering error (memory leak?)
                                                      Adolfo Rozenfeld Adobe Employee

                                                      Normally, you don't want to change the default settings (of course, normally renders don't stall!).

                                                      Let me know if you can upload the project and files to a file hosting service (yousendit, etc).

                                                      • 24. Re: AE rendering error (memory leak?)
                                                        bsnyder175 Community Member

                                                        It's uploading now to a server and should be done by tomorrow morning.  I'll send you a private message with more info about it then.