5 Replies Latest reply on Jul 2, 2013 6:08 AM by Eric T Arroyo

    Finding Bottlenecks

    Eric T Arroyo Level 1

      Hello, I have an interesting issue that I'm hoping someone can shed some light on. First, my system specs.

       

      Intel Core i7-3930k six core (no overclocking yet, just Turbo Boost as I just finished this build a couple days ago).

      16GB RAM (1600Mhz, CS8)

      256GB Sata III SSD for AE Project files, AE disk cache, and source files (two 1080i videos on this project)

      128GB Sata II SSD for OS and Programs (no games, just AE, PrePro, and MS Office)

      500GB Sata II HDD for effect files (explosions, smoke, sound FX, etc. But none of these are used for this project)

      Nvidia GeForce GTX 460 SE (1GB VRAM)

      Windows 7 64 bit

      AE CS5.5

       

      The clip I'm rendering is 10 seconds long and has two layers of 1080i AVCHD videos, 1 layer of rotoscoping for the full 10 seconds, color finesse, keylight, some basic static masks, and I'm exporting to H.264 bluray on my C drive with the Sata II SSD that has just program files and the OS.

       

      Now the issue. My issue is that none of my hardware is maxing out. The CPU bounces around between 50% and 73%, my disk cache and source file Sata III SSD is only transferring at about 10 to 20MB/s (capable of sustained 520MB/s reads and about 505MB/s writes), my Sata II SSD where I'm exporting to is bouncing around at only 2 to 5MB/s, only about 6 of the 12GB of RAM I have reserved for AE multiprocessing is being used, and my video card usage is bouncing around between 40 and 70%. I don't know what else to check. My SSDs are hardly being used and everything else is only reaching about 70% of it's capability.

       

      Does anyone have any idea on how to determine where my actual bottleneck is coming from?

       

      I saw a couple posts from years ago on this, but one ended up being a video card bottleneck and the other was resolved by just changing the output format (likely another video card issue). In my case, the video card's MAX usage for the duration of the render is maybe 72%.

       

      And not that this information is very helpful, but the render time is about 11 to 12 minutes to finish the 10 second clip... It might be useful to know that just loading the 10 second clip fully into RAM for a RAM preview only takes a couple minutes at full res., maybe 3 minutes max.

       

      Thanks in advance for the help!

        • 1. Re: Finding Bottlenecks
          Mylenium Most Valuable Participant

          You are using temporal effects/ features (Rotobrush) that force AE to revert to single threaded processing and you are outputting to a clip format that requires strictly linear progression. You may see better performance by using image sequences, but the part about the effects sticks. Beyond that there's not much to say - never ever will you see full 100% usage on everything. AE takes what it needs and what fits its logic, but nor less and no more. And your RAM preview proibably simply renders faster, because it's at reduced resolution or uses the GPU, which will also not be used for final rendering. Everything perfectly normal. Just get used to it and plan your projects accordingly.

           

          Mylenium

          • 2. Re: Finding Bottlenecks
            Eric T Arroyo Level 1

            Thanks for the reply Mylenium,

            Mylenium wrote:

             

            You are using temporal effects/ features (Rotobrush) that force AE to revert to single threaded processing

             

            I didn't realize Rotobrush was a single threaded process. If that's the case, would this suggest that my bottleneck is that I am only using one core for the Rotobrush and the rest of the cores are waiting for that one to finish? If that's the case that might explain why my GPU keeps bouncing around so much.

             

            Mylenium wrote:

             

            ... you are outputting to a clip format that requires strictly linear progression.

             

            Does that mean AE will only render one frame at a time regardless of whether the "render multiple frames simultaneously" box is checked?

             

            This makes me want to go run some more render tests without using the Rotobrush and using image sequences for an output to see if that changes how the resources are used. I'll try that tonight when I get home. Any other advice in the mean time is certainly welcome though.

            • 3. Re: Finding Bottlenecks
              Eric T Arroyo Level 1

              Okay, so I ran some tests. First. I tried rendering a 6 second scene that has 6 layers of 1080i footage, keylight, and some corner pinning on 5 of the layers.... as well as a wire removal (all pretty simple stuff). I then tried exporting it as a JPEG sequence, h.264, and h.264 blu-ray. This produced significantly different results than my Rotobrush test. In each of these exports the processor was used at 95 to 100% of its potential, the SSD's were used to about 25%, and the RAM had to keep dumping every few frames. So in this case it looks like RAM was one of the bigger bottlenecks because it kept running out and having to swap out to the SSD. Do you agree?

               

              I also ran another test with an Element 3D template that used dozens of Element 3D layers, multiple cameras, motion blur, depth of field, and a few other minor effects. I output to both h.264 and h.264 blu-ray for this test. The results here were 50% CPU usage, 40% RAM usage, and GPU usage that was bouncing between 80% and 0% which tells me that the GPU was likely swapping out data from VRAM to the SSD during those drops. Thus, the GPU was the bottleneck for this render. Does this also sound correct to you?

               

              So those two cases were pretty easy to find the bottleneck in which was nice.

               

              I'm still a bit confused with the Rotobrush project though only because my resource monitor shows all 12 cores staying between 50 and 70%. If 11 of the cores are waiting for 1 core to finish its thing with the Rotobrush what is making them work so hard? Is there another resource monitor I can use other than the stock one that comes with Windows that can show me how much each core is being used?

               

              Thanks again for all the help!

               

              -Eric

              • 4. Re: Finding Bottlenecks
                Mylenium Most Valuable Participant

                Since you don't know the internal thread/ handles allocation, you have no way of verifying whether something works as it should or not. Digging so deep would require a debug build with traceable memory hooks. Simply accept that there will always be such situations in AE - more often than you may think at first. Any temporal operation from time-remapping to time effects to exprerssions with valueAtTime() will delay rendering performance or cause MP rendering to be turned off. As for Rotobrush - the actual build of the underlying vector fields is threaded, but the analysis still needs to progress linearly, hence you may see the performnce spiking arbitrarily.

                 

                Mylenium

                1 person found this helpful
                • 5. Re: Finding Bottlenecks
                  Eric T Arroyo Level 1

                  Thanks for all the really helpful and informative responses Mylenium! Even though I can't "see" the bottleneck when using temporal effects I can at least now assume that it is likely the fact that only a single processor is being used that is slowing things down. Looks like it's time to get back into overclocking .

                   

                  That's about it for me. I've learned a ton in just a day. Thanks again for all the help!

                   

                  -Eric