Not sure what you expect. What you call bottlenecks is normal behavior. File I/O with clip-based formats is mostly linear and while it may take some resources to keep 100 files open and decode them from ProRes, retrieving a single frame from a then to be processed single clip out of your 100 is minor however in terms of network usage and processor usage. Sure, the MediaCaches and disk cache also figure in, at least for the AVCHD stuff and yes, AE verifies File integrity, but it's perfectly normal and expected. Likewise, there's enough threads here on this forum explaining when to use MP and when not, when it will revert to single processes and when not, which effects and features work with it and which ones not and so on. That's simply how AE or some of that other stuff works. You'll have to live with it. There is no way to enforce anything. Once you have converted your footage to image sequences, some of that will improve, though.
Thank you Mylenium.
I guess my question is i don't understand what AE is doing and i'd like to know more. An example is with Nuke, depending on how you write out a single file it may make 1000s of chatty file writes depending on compression and optimisations can be made by writing to a local disc then writing a single frame back to the network. Much faster.
In the case of AE for example, 10% CPU and 12% network utilisation seems the norm. There's plenty of free memory (32GB in all the workstations) and the processing isn't high, i.e. not that much going on at this stage. So either there are processes blocking the task that i'm not aware of, or things aren't be monitored correctly.
If i can get a better understanding of what AE is reading/writing/processing then i can adjust my infrastructure to minimise delays.
I did at one point bring everything to local disc and i still see the same kind of under usage, nothing it being maxed out even remotely.
Are there any better tools to understand the reading/writing patterns that are going on under the hood?
You would have to ask the developers for the super-secret details. Generally AE uses a "lazy" approach, meaning it tries to keep everything under its control as long as it can. Files are kept open instead of closing and re-opening them with every frame, stuff is stored in memory and all that. As a result the program spends a lot of time trying to keep things straight instead of working with the data. That and of course its fixed rendering order. Many operations simply have to wait for the full buffer to be processed since AE can't do scanlines like Nuke or buckets/ tiles as common in 3D programs. This hinders processing efficiancy and also results in the sporadic burst of I/O activity, while most of the time the program is just waiting for something. You realyl can't influence most of that, just try to use more AE instances on separate machines or by using e.g. the BGrender script from AEScripts.