23 Replies Latest reply on Jun 21, 2013 8:57 AM by Nicol Simard

    Multi CUDA rendering in CC

    oliewdj Level 1

      Ok so i decided to put my 460gtz in with the 660ti to see comparable render times with the ppbm6 test

       

      660ti on its own 38 seconds same as cs6

       

      660ti + 480 - 120 seconds

       

      gpuz shows 100% gpu utiilization of 460 and on/off 40/50% on the 660...

       

      so i swapped the cards round into slots, and now i get the revers 100% of 660ti and 40/50% flicking on the 460 but still 2 minutes.

       

      the memory bandwidth of the cards isnt too far off... so i wonder if you need identical cards to increase the render performance not hinder it.

       

      I can disable the second card in nvidia drivers for cuda and only use the 660 and my render times go back to 38 seconds.

       

      now do i compare 285gtx in with the 660?

      i also have a second q6600 machine with a 285 in - i could use both 285s (giving me a 295 really) and see if that improves - but i dont have CC on that machine.

       

      any ideas?

       

      cheers

      Oli

        • 1. Re: Multi CUDA rendering in CC
          Bill Gehrke Most Valuable Participant

          Oli,

           

          I have a GTX 680 running at ~100% and a GTX 780 that is taking up the slack (varying from about 30% to one peak at 90%.  If you are using our PPBM6 MPEG2-DVD with GPU turned on I went from 24 seconds with the GTX 780 only to 16 seconds with both installed.  The GTX 780 is in PCIe slot 1.  I guess later I will switch them  but I would have thought that PCIe slot 1 would have the largest burden.  This is on my older hex-core X58 at 4.2 GHz and 24 GB of RAM.  

          • 2. Re: Multi CUDA rendering in CC
            oliewdj Level 1

            Hi Bill

            yes thats the one - i got to 38 seconds with cs6, and CC replicates the same time with one card which is good consitency.

             

            Your time is great... 16 seconds!

            so i must decide.. is the 460 slowing the 660 down somehow because surely if its 100% on the 660 it will be doing the same performance as before.. unless somehow memory bandwidth or its being throttled?

            thanks

            Oli

            • 3. Re: Multi CUDA rendering in CC
              oliewdj Level 1

              Think ive figuired out my problem.. on the p8p67 board you have to enable the second pci slot for gpu, this in turn disables USB3 and 2 of the pciex slots and then only works at 4X...

              Im getting over 40 seconds with both cards working, i put the 660 in slot 1 @ 16x and the 460 in slot 2 @ 4x and i was around 42 seconds

              i then swapped the 460 to slot 1 @ 16x and the 660 to slot2 @ 4x  still about 48 seconds

               

              so i then disabled the 460 in nvidia cuda and only let premiere use the 660 @ 4x and my render time was 1min 30, guess the bandwidth is the most important!

               

              when i disable the 660 and only use the 460 @ 16x the render time is 62 seconds much like my original ppbm6 tests.

              looks like ill just be using one gpu slot from now on shame....

               

              unless i had a 590 ??? running @ 16X

              • 4. Re: Multi CUDA rendering in CC
                Fabio Pis Level 1

                Hello Bill, nice results.

                Have you gtx 680 result only?

                thank you in advance

                • 5. Re: Multi CUDA rendering in CC
                  Bill Gehrke Most Valuable Participant

                  Not yet Fabio, will try to get that in the next day or two for you. 

                   

                  I have tested, but in all the changing of GPU's it turns out those results (GTX 480, GTX 580, GTX 680 and GTX 780) were all accidently done in a PCIe x8 v2.0 socket therefore they are a few (probably 3-5) seconds high invaladating much work.
                  • 6. Re: Multi CUDA rendering in CC
                    oliewdj Level 1

                    i think i definetly need a dual core gpu to make use of MPE multi cuda rendering due to my motherboard limitations..

                    Bill does your motherboard run both cards at 16X per gpu simultaneously looking at gpuz?

                    Oli

                    • 7. Re: Multi CUDA rendering in CC
                      RjL190365 Level 5

                      Oli,

                       

                      Bill's previous dual-GPU results were done with the second GPU installed in the wrong slot. Most X58 motherboards have three or more PCI-e x16 physical slots, of which only two run at full x16 bandwidth. However, the x16 slots are only PCI-e 2.0 compliant. Plus, the LGA 1366 CPUs do not have the PCI-e hub on the CPU (but instead, the PCI-e 2.0 hub is located on the motherboard, on the IOH that's part of the chipset). Remember, the X58 IOH provides up to 36 PCI-e 2.0 lanes.

                       

                      On the other hand, the LGA 1155 platform (and by extension the LGA 1150 platform) have only 16 available PCI-e lanes - total (not counting any PCI-e lanes provided by the chipset's PCH). Your Asus P8P67 motherboard does not have a x8/x8 switch at all, so the "secondary" PCI-e x16 physical slot is actually run at up to x4 bandwidth off of the P67 PCH instead of the CPU.

                       

                      And a switch to a dual-GPU card will not solve the problem imposed by the mainstream CPU platform: The two GPUs will still share the same 16 lanes of PCI-e bandwidth even with a single card (this means that the two GPUs on a GTX 690 will each receive only eight PCI-e lanes of bandwidth).

                      • 8. Re: Multi CUDA rendering in CC
                        oliewdj Level 1

                        Thanks for that, confirms my suspicions, i guess im limited to one card now unless i upgrade the one card to the highest end i need a higher end platform for more upgradeability and 2X full 16x cards.

                         

                        even then... im pretty happy with the speed im getting now with new card and raid but the dual gpu render was really a teaser !

                         

                        be interesting to somehow benchmark the 16lanes shared on my system.. if i installed both my 285s and did some benchmarking at 16x 4x with both or one that might get to the bottom of it to see how much bandwidth they need?

                         

                        cheers

                        Oli

                        • 9. Re: Multi CUDA rendering in CC
                          RjL190365 Level 5

                          You actually did this earlier with the GTX 660 Ti. It took a minute and a half in the MPEG-2 DVD test in PPBM6 because that GTX 660 Ti is constrained to PCI-e 2.0 x4 bandwidth (compared to the 38 seconds that you got with that same GTX 660 Ti operating in the correct for your CPU PCI-e 2.0 x16 bandwidth). Plus, the fact that your encoding test times got slightly longer with the second card in the PCI-e x4 slot tells you one thing: A system running two discrete GPUs in an imbalanced x16/x4 mode will never deliver as fast of a performance as a single GPU on that same system no matter what.

                          • 10. Re: Multi CUDA rendering in CC
                            Bill Gehrke Most Valuable Participant

                            Oli,

                            oliewdj wrote:

                            Bill does your motherboard run both cards at 16X per gpu simultaneously looking at gpuz?

                            Oli

                             

                            Yes both are running at PCIe x16 version 2, and what I have to do to see the GPU utilization is run two instances of GPU-Z to be able to see them as 16 seconds is not enough time to go back and forth.

                             

                            I am now wondering what could be achieved with version 3.0 slots

                            • 11. Re: Multi CUDA rendering in CC
                              oliewdj Level 1

                              Yes, definetly came to that conclusion today... new chipset / 6 core and multiple gpu i think for the next render pc i think!

                               

                              Bill

                              On game testing 3.0 slots gave 8-10% increase over 2.0....

                              • 12. Re: Multi CUDA rendering in CC
                                Fabio Pis Level 1

                                Thank you Bill,

                                I installed my GTX 580 again to do some testing with PPBM CS5.5 v1 and 3dsmax iray renderer.

                                so

                                 

                                GTX TITAN PCIE x16 3.0

                                GTX 580 PCIE x16 2.0

                                 

                                 

                                ppbm result is  "52","36","36","4" (i could achieve better disk performance if I have time to reorganize it emptying processes and space

                                 

                                gpuz_0.jpg

                                You could observe that cuda on GTX 580 seems not active, but it is working

                                gpuz_1.jpg

                                And these are how gpu's are used during adobe encore cc encoding

                                 

                                Then I share a right utilization of both GPU that I achieve with IRAY renderer for 3ds max

                                gpuz_2.jpg

                                How you can see in this case all gpu are full loaded

                                 

                                Maybe, today I will try a AMD ATI 7970Ghz version, but I think that different products will not work togheter

                                • 13. Re: Multi CUDA rendering in CC
                                  oliewdj Level 1

                                  Would still be instesting to see if say a 590 yields any improvement on a system that can only use 16lanes, test renders with cs6 and then CC.

                                  • 14. Re: Multi CUDA rendering in CC
                                    Nicol Simard Level 1

                                    Am I reading this thread properly and Premiere Pro CC now supports two video cards attached in SLI?

                                    • 15. Re: Multi CUDA rendering in CC
                                      Bill Gehrke Most Valuable Participant

                                      Nicol Simard wrote:

                                       

                                      Am I reading this thread properly and Premiere Pro CC now supports two video cards attached in SLI?

                                      Yes and No

                                       

                                      Adobe Premiere 7.0.0 supports two CUDA GPU's but not connected in SLI

                                      • 16. Re: Multi CUDA rendering in CC
                                        Nicol Simard Level 1

                                        So you just plug in a second card and let things go? No configuring to do? Is there any instructions anywhere regarding how to set this up?

                                        • 17. Re: Multi CUDA rendering in CC
                                          Bill Gehrke Most Valuable Participant

                                          Just Plug-and-Pray!  Hope your power supply is adequate and you have the necessary power cables.

                                          • 18. Re: Multi CUDA rendering in CC
                                            Nicol Simard Level 1

                                            Yep, I've got a 1250 W power supply that came with lots of cables. I will not do it immediately but will probably end up getting a second Titan.

                                            • 19. Re: Multi CUDA rendering in CC
                                              oliewdj Level 1

                                              Nicol - Two titans would be a great setup! just looked at your motherboard spec on the ppbm test - its a monster! 7 PCI express 3.0 x16 slots.

                                              This is extremely interesting for me as my current setup limits me from using dual cards as the pci lanes are limited and without upgrading the motherboard and cpu really i cant use 2 cards. So the best CUDA card singularly would be my maximum speed increase - Titan over 660ti......?.

                                              thanks

                                              • 20. Re: Multi CUDA rendering in CC
                                                Fabio Pis Level 1

                                                Today I did a test with a GTX 690 and Premiere Pro CC in trial way

                                                The results are not good for me.

                                                 

                                                Gpus are used but not in a linear way and min/max usage is very variable from 10% to 80% for both gpus

                                                Nvidia driver i setted to SLI (multigpu mode), I have used DNxHD codec to export from media encoder cc

                                                In this config it seems to work better

                                                 

                                                If I disable multigpu all gpus are working but less used than before (multigpu selected)

                                                Intel chipset is X58 and cpu is I7 980x

                                                 

                                                Inside this system there is also a gt 640 gfx  used only for phisyque.

                                                If I enable it for cuda ,system uses it a 90% and the two 690 gpus at a minimal value (and not stable value ) of 10/15% usage

                                                • 21. Re: Multi CUDA rendering in CC
                                                  oliewdj Level 1

                                                  Thanks Fabio,

                                                  This is all interesting as your x58 will have 36 pci lanes.. so i thought it would be much faster using both gpus?

                                                  Have you done the ppbm6 benchmark on the mpeg2 export to gague a real difference enabling one and disabling.

                                                   

                                                  thanks

                                                  Oli

                                                  • 22. Re: Multi CUDA rendering in CC
                                                    Fabio Pis Level 1

                                                    sorry oliewdj, I have used a pc of a friend of mine and in trial mode some codec are not available (eg mpeg-dvd)

                                                    The interesting things is that premiere pro CC seems to work in sli mode (multigpu activated from driver) and in normal dual card mode (no multigpu/sli).

                                                    But I think that GPU's are not used very well in stable full load

                                                    • 23. Re: Multi CUDA rendering in CC
                                                      Nicol Simard Level 1

                                                      I plan to do the test in about 3 or 4 weeks. I have some projects I want to finish before I start fiddling with my computer.