3 Replies Latest reply on May 8, 2013 7:29 AM by Mylenium

    Performance (Network and MP) with AE

    paulinventome Level 1

      I've read through a lot of threads, and seem to remember seeing some that i've not been able to find again, i'm hoping i can get some pointers overall and hopefully also contribute back.


      I'm using AE to process a lot of footage for a feature. My other tool sets include Nuke and Modo, but AE is useful because it's scriptable and the dark energy plug in works with it. I'm also having to use it to fix some camera hardware issues too. So it's been really good for me. However when dealing with 10,000s frames any performance tweaks can make a big difference for me


      My observations/questions are:


      I'm running a few different workstations and dedicated render nodes around shared storage. At the moment the shared storage is around a synology 1812+ and 513 NAS over gigabit ethernet, i can't quite run to a fibre SAN, however i am looking at a bigger NAS (the 3612xs which offers 4xGigE and 2x10GigE).


      One observation is that i have my processing split into different AE projects and each project links to about 50-100 source ProRes and AVCHD files, each project takes around 10 minutes to open and i think that's really slow. The project and the source files are on the NAS. I render the results to local disc, that sped things about about 20% so the network storage is just reading. My question is what is AE doing with these source QT movies that takes so long to open, is it scanning them through or something?


      One output of my pipeline will probably be 2K OpenEXR files, and i remember reading some big caveats about rendering file based with MP in that AE creates a storm of reads/writes. I haven't been able to find this thread again?


      My CPU utilisation also seems quite low and my network is around 12%. In other words i'm not sure what the bottle necks are or how to go about finding them. Whether there are tools out there to anaylse network traffic - is it bandwidth? latency? I've tweaked many MP settings, so each thread has lots of ram, running 2 to 4 cpus but never managed to saturate some aspect of the processing. I just don't know what is taking the time. In this first stage files are being written as QT and i know that QT has a huge overhead in terms of writing (compared to AVI for example). But again, techniques for understanding and troubleshooting this would be useful to hear?


      So any pointers would be greatfully receieved!




        • 1. Re: Performance (Network and MP) with AE
          Mylenium Most Valuable Participant

          Not sure what you expect. What you call bottlenecks is normal behavior. File I/O with clip-based formats is mostly linear and while it may take some resources to keep 100 files open and decode them from ProRes, retrieving a single frame from a then to be processed single clip out of your 100 is minor however in terms of network usage and processor usage. Sure, the MediaCaches and disk cache also figure in, at least for the AVCHD stuff and yes, AE verifies File integrity, but it's perfectly normal and expected. Likewise, there's enough threads here on this forum explaining when to use MP and when not, when it will revert to single processes and when not, which effects and features work with it and which ones not and so on. That's simply how AE or some of that other stuff works. You'll have to live with it. There is no way to enforce anything. Once you have converted your footage to image sequences, some of that will improve, though.



          • 2. Re: Performance (Network and MP) with AE
            paulinventome Level 1

            Thank you Mylenium.


            I guess my question is i don't understand what AE is doing and i'd like to know more. An example is with Nuke, depending on how you write out a single file it may make 1000s of chatty file writes depending on compression and optimisations can be made by writing to a local disc then writing a single frame back to the network. Much faster.


            In the case of AE for example, 10% CPU and 12% network utilisation seems the norm. There's plenty of free memory (32GB in all the workstations) and the processing isn't high, i.e. not that much going on at this stage. So either there are processes blocking the task that i'm not aware of, or things aren't be monitored correctly.


            If i can get a better understanding of what AE is reading/writing/processing then i can adjust my infrastructure to minimise delays.


            I did at one point bring everything to local disc and i still see the same kind of under usage, nothing it being maxed out even remotely.


            Are there any better tools to understand the reading/writing patterns that are going on under the hood?


            many thanks


            • 3. Re: Performance (Network and MP) with AE
              Mylenium Most Valuable Participant

              You would have to ask the developers for the super-secret details. Generally AE uses a "lazy" approach, meaning it tries to keep everything under its control as long as it can. Files are kept open instead of closing and re-opening them with every frame, stuff is stored in memory and all that. As a result the program spends a lot of time trying to keep things straight instead of working with the data. That and of course its fixed rendering order. Many operations simply have to wait for the full buffer to be processed since AE can't do scanlines like Nuke or buckets/ tiles as common in 3D programs. This hinders processing efficiancy and also results in the sporadic burst of I/O activity, while most of the time the program is just waiting for something. You realyl can't influence most of that, just try to use more AE instances on separate machines or by using e.g. the BGrender script from AEScripts.