7 Replies Latest reply on Feb 29, 2016 12:28 PM by Laundry_Design

    Issues with AE CC15 tasks being completed on Render Farm

    Laundry_Design Level 1

      Hello,

       

      We are using Thinkbox's Deadline to submit jobs to our 30 node render farm and are having issues with random nodes (different nodes each time) never resolving their tasks. We then have to manually stop the job and resubmit in order for the render to start happening. Other times it will hang on 0% for several minutes but suddenly jump to 100% and show the job as complete. However, when we check the folder, there will be several frames missing. I spent an entire day with Thinkbox support on this issue and they came to the conclusion that it is an issue outside of Deadline; then told me I should get in touch with Adobe support. I have several screenshots, job logs, and slave logs, as well as the thread of emails and command line prompts with strange results from Thinkbox for reference if these will help. Below is a screenshot of some render nodes that have been at 0% for nearly 14 hours. Thank you.

       

      Screen Shot 2016-02-05 at 11.00.24 AM.jpg

        • 1. Re: Issues with AE CC15 tasks being completed on Render Farm
          Szalam Adobe Community Professional & MVP

          This is just a user-to-user forum, by the way. Occasionally Adobe employees pop in here, but, if you want official Adobe support, you'll need to contact them. Contact Customer Care

           

          What version of AE do you have on each node? Have they all been updated to 13.7.0?

          What OS are all of the nodes using?

          What third-party plugins are involved in the project?

          • 2. Re: Issues with AE CC15 tasks being completed on Render Farm
            Laundry_Design Level 1

            Yes sir, thank you. I have contacted Adobe support several times and they report that they are unable to help me with issues regarding render farms, but to go to forums.adobe.com and post a question. I'm refraining from posting my true feelings on that matter, and hoping this forum may actually help.

             

             

            AE CC2015 v13.7.0

            Windows 7

            about 7 or 8 plugins, all installed on the farm as well, usually plugin errors will be noted in the deadline monitor. With this issue there are no errors being reported at all. If you need, I can get a complete list of the plugins being used from the artist and report them to you.

            • 3. Re: Issues with AE CC15 tasks being completed on Render Farm
              James Pageau Level 1

              I have seen this issue too.   I have found it does not happen if I keep Deadline to one task per machine.  But that does not take advantage of our cores very well.  I also found that the nodes were trying to autosave to the AE file at the same time causing conflicts.  I ended up going into each node launching AE and turning off Autosave every 20 minutes and "Save when starting render queue".   I also disabled the disk cache preference on the render nodes as I think it does an extensive check of all the resources linked to the AE file that was very demanding on our network and NAS when all the nodes start up.  I think some nodes were timing out and not recovering updating the cache and I did not want them to used cached data anyway.


              We are usually working with OpenEXRs for input and output on our jobs and are currently running AE 2015 13.7.2. 

              Are you running any 3rd party filters?

              • 4. Re: Issues with AE CC15 tasks being completed on Render Farm
                James Pageau Level 1

                Also Some of the latest RedGiant plugins like Colorista III  will crash a render node if it does not have a GPU, as they decided to take away the CPU pathway for rendering.  RG did not do very good error handling with this, so the installer does not warn you and AE Render just quits if it tries to run that plugin.  It was a big change that was not communicated very well by Red Giant, especially since many farms don't have GPUs.   I am not sure if Deadline showed that as a failed or just hung when that happened.  But that might be another thing to consider.

                • 5. Re: Issues with AE CC15 tasks being completed on Render Farm
                  Laundry_Design Level 1

                  Thanks James. We noticed it is a problem when adding more than one task as well. I'm going through each node now,  turning off the auto-save and disabling disk-cache. We have Colorista III installed but are not using it on this project. I wonder if it would still be causing a problem though. If the problem persists, I will try uninstalling that as well. As far as 3rd party filters, I don't know. I know we have a few plugins involved and I'm waiting to see if I can get a list from the artists. In the meantime, I will try the above. Thanks for the tips. I'll let you know how it goes.

                   

                  Have you still be having issues with assigning more than one task after taking these steps?

                  • 6. Re: Issues with AE CC15 tasks being completed on Render Farm
                    James Pageau Level 1

                    I think it had a tendency to say it rendered a task without actually outputting frames intermittently if we had more than one task per workstation.  I went with what seemed to be more reliable.  I am not sure if the problem is completely gone our just more rare this way.  (We are more constrained by network bandwidth than CPU power on our AE renders so typically I put 5 to 10 nodes on our AE composting jobs.  The real reason we have a farm is for rendering in Max which makes much better use of our cores and is less demanding on the network.)

                    • 7. Re: Issues with AE CC15 tasks being completed on Render Farm
                      Laundry_Design Level 1

                      After many reinstalls and tests, I was able to produce a dialogue box on the render node that indicated a crash. This only happened, as you stated, when setting it to 3 or more concurrent tasks. Once the dialogue box is closed, thus closing the program, Deadline will register an error and kick the render to another node. This will usually do the trick and finish the job. But the program must manually be closed. Maybe someone out there knows a setting or hack that will just bypass this dialogue box and automatically close the program so Deadline can kick to another node and get on with it?Screen Shot 2016-02-28 at 8.16.17 AM.png