38 Replies Latest reply on Oct 27, 2016 4:54 AM by Dustin_Farrell

    X99-E WS (multi GPU + TBoltExII Card) issues

    rsukhram@tgioa.com Level 1

      Good morning,>

      So recently I purchased a ton of gear to get into the x99/DDR4 platform.  I have 3 GPUs from my previous build but cannot get 2 of them to work if the TBolt Card is installed.  >


      The LCD on the board says 95 then restarts over and over ..never going past 95.  Asus Tech support says that means there are not enough lanes to support my needs.  I am using an i7-5960x CPU.  I have 64 GB G.Skill ram.  >


      My previous build (Z9PE-D8 WS board) ran all 3 GPUs just fine.>


      The GPU's I have are the quadro 4000 + quadro K2000 + tesla C2075.  >


      Any thoughts as to what I can do to get this to work?  I am on Bios version 1003.  I would hate to have gone this far to be limited due to the TBoltEx card eating up tons of lanes.

        • 1. Re: X99-E WS (multi GPU + TBoltExII Card) issues
          cc_merchant Level 4

          Why not sell off those cards and replace them with two GTX 980's. That should leave you sufficient PCIe lanes and be much faster than what you have now.

          • 2. Re: X99-E WS (multi GPU + TBoltExII Card) issues
            rsukhram@tgioa.com Level 1

            I am also using a Samsung XP941 (which could also be eating up some lanes).  I was actually considering using 1 Titan-X and rebuilding my dual xeon (with the QuaDRO 4000 + Tesla C2075) as a render only machine.  Not sure how I would set that up tho.

             

            Is there a way to see how many lanes are being used by a device?

            • 3. Re: X99-E WS (multi GPU + TBoltExII Card) issues
              ECBowen Most Valuable Participant

              There are some bios settings that have to be done to get more than 1 GPU to work with the TB card. Also you have to have the latest bios version which you do have. The TB card has to be in slot 2. You have to setup the bios correctly with just 1 GPU and then add the 2nd GPU once you have those settings in so the resource charts build around 2 cards. The 3rd card probably wont happen without an expansion chassis because of the slot spacing in most ATX cases.

               

              Eric

              ADK

              • 4. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                rsukhram@tgioa.com Level 1

                I currently have 1 GPU (Quadro 4000) in slot 1.  The TBolt card is in slot 2.

                 

                Would you happen to know what those settings should be?  My 2nd GPU will be the Tesla C2075 until i replace all my GPUS with the Titan-X.

                 

                Also note that I have a CM Cosmos II chasis so space is not an issue.

                 

                thanks ECB

                • 5. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                  ECBowen Most Valuable Participant

                  I posted them in the red forums. I am pasting them below.

                   

                  Try the following:

                  Pull TB card but leave cable plugged in.

                  Go into the bios

                  Go to advanced mode

                  Go to the boot tab

                  Disable fast boot

                  Int19 trap response - change to postponed

                  Above 4G Decoding - Change to enabled.

                  Save and exit.

                   

                  Let the systemboard build the resource charts again which may take several reboots shortly after post. Then make sure it boots to the OS. Shut the system down and then put in the TB card. If it still doesnt work take the card out and disable AIC support totally. Then try it again. As long as you have the security cable plugged in it should still give you the TB card setting options. Also You may have to disable the Sata express port to free up resources. I did that here on a WS to get the latest bios to work with 2 980GTX cards and the Thunderbolt card with a Xeon chip

                   

                  Eric

                  ADK

                  • 6. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                    rsukhram@tgioa.com Level 1

                    Try the following:

                    Pull TB card but leave cable plugged in.

                        you mean the 9-pin to 5-pin cable?

                     

                    Let the systemboard build the resource charts again which may take several reboots shortly after post.

                         is there something I should be looking at to confirm this?

                     

                    Then make sure it boots to the OS. Shut the system down and then put in the TB card. If it still doesnt work take the card out and disable AIC support totally.

                         where is AIC support?

                     

                    Then try it again. As long as you have the security cable plugged in it should still give you the TB card setting options. Also You may have to disable the Sata express port to free up resources.

                         Would this affect my M.2 XP941 drive...which happens to be my Main OS drive.

                    • 7. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                      Bill Gehrke Adobe Community Professional & MVP

                      rsukhram@tgioa.com wrote:


                      Is there a way to see how many lanes are being used by a device?

                      If you run GPU-Z you can see how many lanes each GPU is actually using when they are loaded.  You can actually run multiple instances of GPU-Z for each GPU card installed.  I also am having problems with my X99-E WS living up to its specifications even without a ThunderBolt card (three GPU's) and a SM951 M.2 SSD installed.

                      • 8. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                        rsukhram@tgioa.com Level 1

                        gotcha Bill.  Im also a bit disappointed at the moment.  "Thoughtful" on the Asus forums mentioned something about other Asus boards having a TB reference :

                         

                        I do not see where support for Thunderbolt has been added to the microcode for the ASUS X99-E WS motherboard.

                         

                        Support for Thunderbolt was specifically revised and listed for the other ASUS X99 motherboards:

                        *X99-DELUXE; BIOS Version 1004; 10/23/2104

                        *X99-PRO; BIOS Version 1004; 10/23/2014

                        *X99-A; BIOS Version 1004; 10/23/2014

                         

                        Hopefully, ASUS will add specific support for their ASUS ThunderBoltEX II card (or Thunderbot, in general) in a future BIOS update for their ASUS X99-E WS motherboard.

                         

                        I wonder if there will be conflicts between the Thunderbolt controller, SATA Express controller, PCI Express with NVMe protocols, GPU, SATA controllers, etc., due to lack of available lanes. We need guidance in the form of a component matrix to help clients to better and successfully populate ASUS motherboards with old and newly emerging hardware.

                         

                         

                        My storage decision (the Cineraid) depends on the TB card which limits my GPU options + using the M.2 dive....SMH non-stop but trying to be positive and was thinking about offsetting all my rendering needs to my previous build.

                         

                        I started a new thread here.  Let me know if you think this makes sense to do.

                        https://forums.adobe.com/thread/1821493

                        • 9. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                          ECBowen Most Valuable Participant

                          We ship the WS board with the TB cards. The latest bios works fine for recognizing the cards.

                           

                          The cable I am referencing in the post above is the security header cable that connects the TB card to the board. That cable has to remain plugged in for the TB card to work and the settings in the bios to show up. They wont show if you disconnect that cable. So what you have to do is remove the card from the slot and lay it down on the bench with the system off but keep that cable plugged into the card and board. Then when you boot up and go into the bios the Thunderbolt controller options will be there under the advanced tab in advanced mode. AIC support is listed under those options.

                           

                          The Resource charts build automatically by the board. You don't see or do anything. It just will restart shortly after you turn it on as it rebuilds those resources.

                           

                          Yes the Samsung drive would use the Sata Express controller. You may not have to disable that to free enough resources for those cards. I did here because it was not being used.

                           

                           

                          Eric

                          ADK

                          • 10. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                            rsukhram@tgioa.com Level 1

                            ok noted.  I am running a few VDP jobs at the moment so I cannot bring the system down but first chance I get I will try your settings.  Thanks for all the info!

                            • 11. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                              dcwoodsdesign

                              This is a known issue.  I have the same problem with 2 980 Ti's (yes brand spankin new).  I've tried 2 980's also and it seems to be a problem with the GPU UEFI based on dozens of other forum posts.  Funny thing is I've also tested with 2 Titan Blacks and the Titans work fine.  I was on the phone with both customer tech support from Asus and EVGA and both blamed it on each other.  After EXTENSIVE research it is a problem with both.  Asus BIOS hasn't dealt with modern GPU UEFI's (newer GPUs fail to show up on their "qualified vendor list" for their flagship X99 boards, and the GPU UEFI if disabled can lead to a successful TB2 card boot.  I've been struggling with this for months now but have not contacted EVGA to figure out what can be done with the GPU UEFI.  Ever get any more info than what I mentioned?  I would love to have closure on this!

                              • 12. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                rsukhram@tgioa.com Level 1

                                Sorry I can't give closure on this.  My config now has a single Titan-X with the TBolt card + M.2 for primary.  

                                • 13. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                  astross

                                  I have a single Titan-x with Thunderbolt card but I have had no luck getting to to run.  What is your slot configuration?

                                  • 14. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                    rsukhram@tgioa.com Level 1

                                    I attached 2 pics for you to reference.  TBolt in Slot 2,  Titan-X in Slot 3.

                                     

                                    IMG_0186.JPG

                                     

                                    IMG_0185.JPG

                                    • 15. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                      rsukhram@tgioa.com Level 1

                                      just curious if you are still experiencing this issue?  My system has been running (not to my total expectations) but running nonetheless so I have not tried updating any drivers or firmwares since.

                                       

                                      Also do any of you have an issue where the NIC card does not connect to your domain unless your disable and enable?

                                      • 16. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                        jamess55428008

                                        I'm having the same issues you are having.  Same MOBO, same ThunderboltEx II card and I'm running a single GTX 980 (just for the install, have a second one waiting to get hooked up). 

                                         

                                        I'm losing my MIND! 

                                         

                                        How did you end up finding these advanced menu settings in the BIOS?  I've followed everything listed above to no avail.  I see some of the menu options in BIOS but nothing that lists as Thunderbolt, nor the AIC options either...

                                         

                                        For your hardware installation, did you have to connect the included displayport cable back to the Titan X in order for the Thunderbolt card to work?  Very confusing indeed considering we don't have any DisplayPort outputs on our specific MOBO model. 

                                         

                                        To recap:

                                         

                                        I first updated the BIOS (just released last month, July 2015)

                                        Then I installed the driver from the included disc

                                        Then powered down, installed the card in the second slot, and connected everything but the displayport cable.

                                        Booted up and no sign of thunderbolt in the device manager, just a bunch of listings for "base system device" 

                                         

                                        I've since tried the bottom most slot but no luck either. 

                                         

                                        Please tell me exactly how you configured your system to work.  I'm gonna be losing sleep over this all weekend otherwise.

                                         

                                        Thanks in advance!

                                        • 17. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                          rsukhram@tgioa.com Level 1

                                          James,

                                           

                                          I also had difficulty in the beginning and to be honest I am not sure if any of my BIOS versions opened up the TBolt menu but I do see one in my BIOS.  Also I use my TBolt card strictly to hook up an external Raid Storage device so I have not tried using the DisplayPort.  From a hardware side make you after you connect the TB card that you connect the ribbon.   I would advise that you use the drivers from the Asus site and not the disc since those are going to be more up to date.   Also if I am not mistaken the manual for the TB card says not to have the device installed when installing the drivers.

                                           

                                          I took some pics of my BIOS settings that may be of some help.  Note on my 2nd pic along the tool bar there is an "Advanced" option.  In there is where I found "Intel(R) ThunderBolt"

                                           

                                          IMAG1249.jpg

                                           

                                          IMAG1250.jpg

                                           

                                          IMAG1253.jpg

                                           

                                          IMAG1254.jpg

                                           

                                          IMAG1255.jpg

                                          • 18. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                            rsukhram@tgioa.com Level 1

                                            Also here is my device manager screenshot.  I also had a lot of unresolved base system drivers.  Install the Chipset driver again.  That worked for me.

                                             

                                            777.PNG

                                             

                                            888.PNG

                                            • 19. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                              jasonwd

                                              I'm so glad you guys started talking about this...

                                               

                                              I have an X99 Deluxe with a 5960K and a pair of 970s - built for 3D rendering grunt in February

                                               

                                              When I built the system I messed with it no end and couldn't get the stupid thing to work, so it has been sitting there ever since, blocking air to the top GPU.

                                               

                                              I just updated the BIOS and all the drivers for Win10 and, thanks to you guys' advice I feel metalled to have another go at making it work.

                                              • 20. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                dcwoodsdesign Level 1

                                                If you follow ECBowmen's advice, you'll get it to work.  I had almost the same setup and I got it to work.  I upgraded to an Asus X99 Sabertooth after a failure on the X99 Deluxe (thank god for Microcenter's warranty/return policy) and the same problem exists on that board.  Got it to work eventually on that too.  Best of luck!

                                                • 21. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                  rsukhram@tgioa.com Level 1

                                                  Anyone with this config install Windows 10 yet?  Im more concerned about the TBolt adapter since it connects to my main Raid 5 enclosure.

                                                  • 22. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                    rsukhram@tgioa.com Level 1

                                                    FYI I just did a fresh install (via the UEFI method and a DVD) and used the latest drivers available for the TBoltII card and Areca Raid enclosure (which is only win 8.1) and got everything to work with no issues.

                                                    • 23. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                      Irenabys

                                                      I have finally had success with the following configuration with Thunderbolt display routing and thunderbolt drive access in both Windows 10 and hackintosh OS X 10.10 (not no hot swapping in OS X)

                                                      ASUS X99-DELUXE BIOS Version 1901

                                                      Intel Core i7 5930K

                                                      EVGA GeForce GTX 970 4GB

                                                      ThunderboltEX II Card

                                                      Clover Bootloader

                                                      Windows 10 / Hackintosh OSX 10.10 Yosemite

                                                      In addition to the comments in this thread I found Josh Olsons description at the link below to be most useful and have added my install steps on stack exchange at:

                                                      bios - Thunderbolt on PC on an X99 motherboard (and the R9 295x2)? - Super User

                                                       

                                                      Thanks to everyone who contributed system info here.

                                                      • 24. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                        rsukhram@tgioa.com Level 1

                                                        thanks for the share and sorry for this late response.  For some reason I am not getting notifications from Adobe about new messages.

                                                         

                                                        Good luck with the build!

                                                        • 25. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                          jasonwd Level 1

                                                          Irenabys I'd like to know how you got your Hackintosh working seeing as your machine is pretty similar to mine... was it hard?

                                                           

                                                           

                                                          I have given up with the stupid Thunderbolt card and is now out of my machine and sitting in a box.

                                                          • 26. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                            Irenabys Level 1

                                                            Don’t give up on the thunderbolt - get it running in windows first. Follow the other posters advice. Use 1 GPU to start with and see the details in the link I posted. Let me know if you need my bios settings however its all about getting the motherboard to locate the card.

                                                             

                                                            I have a very stable hackintosh on the ASUS Deluxe as described. I’ve had it running days at a time with multiple instances of parallels vms inside it even. I’m running OSX Yosemite and followed to the letter the instructions at http://www.rampagedev.com I have had no success upgrading to El Capitan from my copy or a clean install but am pretty content: Thunderbolt, apple bluetooth mouse and (version 1) blue tooth trackpad all work (incidentally the blue tooth drivers allow the mac mouse to work on windows 10 which is great too).

                                                             

                                                            Good luck

                                                            • 27. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                              Forest Hightower

                                                              I am planning to allocate funds for a x99 system myself, mine is going to be only with 1 gpu at this stage, mainly for Adobe Premier / After Effects, with some Cinema 4d work if I can.

                                                              I am considering Asus motherboards because of Thunderbolt, what do you guys recommend at this stage? also for the GPU I realize A lot of people going for 980ti, but also there is this PNY Quadro m4000 card that some folks suggested for my budget, I am wondering if I will have any issues with a TB card?

                                                              • 28. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                josterday

                                                                I have an X99-E WS board with a pair of Titan-X's and still can't get my EXII Card to function in Windows 10:

                                                                 

                                                                - in Slot 2 with single Titan in Slot 3

                                                                - BIOS Recognizes the card.  Everything is set to what rsukhram and ecbowen showed.

                                                                - The MB header and Display Port cables are both connected.

                                                                - Tried 8.1 and W10 Thunderbolt Drivers

                                                                - Tried installing card before and after fresh Driver installs

                                                                - Tried installing fresh Chipset Drivers

                                                                 

                                                                After installing and running the Thunderbolt Software Package I tried connecting various different devices but none of them show in the Device Manager.  My BMD Ultrastudio DOES however power up when you connect the TB cable, so I know the card is providing power.

                                                                Thanks for any advice.

                                                                • 29. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                  Forest Hightower Level 1

                                                                  I will be trying ASRock mobo extreme now to test their Thunderbolt Adapter instead of Asus...

                                                                  • 30. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                    rsukhram@tgioa.com Level 1

                                                                    note that for me I had to install drivers from my CineRaid manufacturer's website for the device to work properly.  not sure if the same will make a difference for you.

                                                                    • 31. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                      Forest Hightower Level 1

                                                                      Really? for the ASRock mobo you had to do that?

                                                                      • 32. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                        johnb51371938

                                                                        I'm strugglng getting this to work too...

                                                                        I ve tried pulling out the TB card from the slot and leaving the cable plug in the mobo and the card but still can get the intel thunderbolt option in the bios to show up.

                                                                        Any ideas ?

                                                                         

                                                                        My rig is Asusx99 ws bios 2006

                                                                        64 gb ram

                                                                        5960x

                                                                        dual titan x

                                                                        • 33. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                          ECBowen Most Valuable Participant

                                                                          You can't get the Thunderbolt settings to show under advanced in the bios?


                                                                          Eric

                                                                          ADK

                                                                          • 34. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                            johnb51371938 Level 1

                                                                            Hi !

                                                                            Yep... i leave the cable plug in at both end, but pulled out the card, it doesn't show up in advanced.

                                                                            Obviously when i boot with only one GPU, and the card in the slot i ve got the option.

                                                                            • 35. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                              ECBowen Most Valuable Participant

                                                                              Try just running the TB2 card in the slot 2 with 1 video card in slot 3. Then go into the bios and hit F7 for advanced mode. Go to the Boot tab and disable fast boot. Then change INT 19 trap response to postponed and enable Above 4G decoding. Then go to the advanced tab and then intel Thunderbolt configuration. Make sure you have TBT reserved I/O support enabled and the Reserved I/O per slot change from 4 to 8. Then save and exit from the bios and let it boot to Windows. Shut it down and then put the 2nd video card into slot 5. See if that works.

                                                                               

                                                                              Eric

                                                                              ADK

                                                                              1 person found this helpful
                                                                              • 36. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                                gsalucci

                                                                                Hey guys. I have tried all methods listed here and can't get 2x 980's and a Thunderbolt card to work with the x99-E WS. The new BIOS  (3201) actually causes the ThunderboltEX II to stop working completely, so I had to revert back to 3101.

                                                                                 

                                                                                Some other things to note:

                                                                                In all the "3" versions of the BIOS, delayed INT19 response option is gone completely.

                                                                                "Above 4g Decoding" will cause OSX to not boot if you're using Hackintosh with this board, so enabling it isn't really an option. However, I did try doing this and booting into windows, and the card STILL doesn't work.

                                                                                 

                                                                                Card works perfectly fine with 1 GPU.

                                                                                 

                                                                                After rebuilding the resource charts as instructed, the machine will boot, but the card does not function.

                                                                                 

                                                                                Anyone have any idea on an update to this issue? Thanks

                                                                                • 37. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                                  jamess55428008 Level 1

                                                                                  For what it's worth, gsalucci we were running an identical 980 setup over on our end... never made the jump to 3101, still on 1302 (I believe, if I'm not mistaken).  However, we ran into an incredible amount of issues with the dual 980 setup after premiere updated to 2015.3 and 2015.3 (10.4).  Everything was great until that software bump, but ever since we've actually had to sell off our 980s and adopt a single Titan X (2015, not Pascal, pascal has many issues with Premiere 2015.3 10.4 and 2015.3.  We're running both separately in different suites to test them, and last years titan is thus far the most stable GPU we've tested with our configuration.  It's pricey, but it's worth it.

                                                                                   

                                                                                  I realize you never mentioned premiere functionality with regard to the dual gpus, but I figured it might help to save you some time and headache.  Even if you manage to get the dual cards working, the latest premiere updates and nvidia drivers don't seem to be well matched at present. 

                                                                                   

                                                                                  So long story short, I don't think you have to ditch the gtx 980s.  If you're willing (and this is even possible) I believe you can flash your bios back to 1302, which I believe is what we're running.  Instructions on how to do that should be online, but it sounds like you're comfortable with flashing the bios. 

                                                                                   

                                                                                  If you can get it to 1302, and you run through the same procedures and tick the right settings in the bios from ECBowen (aware you've done this many times before, but this should work under this bios rev. as it has for us for over a year. 

                                                                                   

                                                                                  From there, if you stick to Premiere Pro CC 2015.2 and manually install a legacy recommended certified (not beta) drivers from anytime before Mid-April, you should be right as rain, and not be out a single penny.  Also worth noting, if you are using an SLI bridge linking the two cards, ditch it for sure.  We experimented with that on our setup last year and it caused nothing but issues, all across the spectrum from program issues, to monitor glitches, to boot issues.  No performance boost for premiere, only managed to introduce glitches.  We weren't hopeful, merely curious and die-hard pc tweakers, but it wasn't worth the headache or the time to experiment with it. 

                                                                                   

                                                                                  Hope this helps, and you get your system back online and operational soon! 

                                                                                   

                                                                                  Let me know if you want me to expand on anything. 

                                                                                   

                                                                                  Cheers!

                                                                                  • 38. Re: X99-E WS (multi GPU + TBoltExII Card) issues
                                                                                    Dustin_Farrell

                                                                                    I worked on this for around 12 straight hours and finally found this post. Now I can sleep. EX3 card works with GPU expansion box now. Thank you! I owe you a beer if you are ever in Phoenix.