This is just a user-to-user forum, by the way. Occasionally Adobe employees pop in here, but, if you want official Adobe support, you'll need to contact them. Contact Customer Care
What version of AE do you have on each node? Have they all been updated to 13.7.0?
What OS are all of the nodes using?
What third-party plugins are involved in the project?
Yes sir, thank you. I have contacted Adobe support several times and they report that they are unable to help me with issues regarding render farms, but to go to forums.adobe.com and post a question. I'm refraining from posting my true feelings on that matter, and hoping this forum may actually help.
AE CC2015 v13.7.0
about 7 or 8 plugins, all installed on the farm as well, usually plugin errors will be noted in the deadline monitor. With this issue there are no errors being reported at all. If you need, I can get a complete list of the plugins being used from the artist and report them to you.
I have seen this issue too. I have found it does not happen if I keep Deadline to one task per machine. But that does not take advantage of our cores very well. I also found that the nodes were trying to autosave to the AE file at the same time causing conflicts. I ended up going into each node launching AE and turning off Autosave every 20 minutes and "Save when starting render queue". I also disabled the disk cache preference on the render nodes as I think it does an extensive check of all the resources linked to the AE file that was very demanding on our network and NAS when all the nodes start up. I think some nodes were timing out and not recovering updating the cache and I did not want them to used cached data anyway.
We are usually working with OpenEXRs for input and output on our jobs and are currently running AE 2015 13.7.2.
Are you running any 3rd party filters?
Also Some of the latest RedGiant plugins like Colorista III will crash a render node if it does not have a GPU, as they decided to take away the CPU pathway for rendering. RG did not do very good error handling with this, so the installer does not warn you and AE Render just quits if it tries to run that plugin. It was a big change that was not communicated very well by Red Giant, especially since many farms don't have GPUs. I am not sure if Deadline showed that as a failed or just hung when that happened. But that might be another thing to consider.
Thanks James. We noticed it is a problem when adding more than one task as well. I'm going through each node now, turning off the auto-save and disabling disk-cache. We have Colorista III installed but are not using it on this project. I wonder if it would still be causing a problem though. If the problem persists, I will try uninstalling that as well. As far as 3rd party filters, I don't know. I know we have a few plugins involved and I'm waiting to see if I can get a list from the artists. In the meantime, I will try the above. Thanks for the tips. I'll let you know how it goes.
Have you still be having issues with assigning more than one task after taking these steps?
I think it had a tendency to say it rendered a task without actually outputting frames intermittently if we had more than one task per workstation. I went with what seemed to be more reliable. I am not sure if the problem is completely gone our just more rare this way. (We are more constrained by network bandwidth than CPU power on our AE renders so typically I put 5 to 10 nodes on our AE composting jobs. The real reason we have a farm is for rendering in Max which makes much better use of our cores and is less demanding on the network.)
After many reinstalls and tests, I was able to produce a dialogue box on the render node that indicated a crash. This only happened, as you stated, when setting it to 3 or more concurrent tasks. Once the dialogue box is closed, thus closing the program, Deadline will register an error and kick the render to another node. This will usually do the trick and finish the job. But the program must manually be closed. Maybe someone out there knows a setting or hack that will just bypass this dialogue box and automatically close the program so Deadline can kick to another node and get on with it?