X99-E WS (multi GPU + TBoltExII Card) issues

[Ren Sukhram]

Good morning,>

So recently I purchased a ton of gear to get into the x99/DDR4 platform.  I have 3 GPUs from my previous build but cannot get 2 of them to work if the TBolt Card is installed.  >

The LCD on the board says 95 then restarts over and over ..never going past 95.  Asus Tech support says that means there are not enough lanes to support my needs.  I am using an i7-5960x CPU.  I have 64 GB G.Skill ram.  >

My previous build (Z9PE-D8 WS board) ran all 3 GPUs just fine.>

The GPU's I have are the quadro 4000 + quadro K2000 + tesla C2075.  >

Any thoughts as to what I can do to get this to work?  I am on Bios version 1003.  I would hate to have gone this far to be limited due to the TBoltEx card eating up tons of lanes.

[cc_merchant]

Why not sell off those cards and replace them with two GTX 980's. That should leave you sufficient PCIe lanes and be much faster than what you have now.

[Ren Sukhram]

I am also using a Samsung XP941 (which could also be eating up some lanes).  I was actually considering using 1 Titan-X and rebuilding my dual xeon (with the QuaDRO 4000 + Tesla C2075) as a render only machine.  Not sure how I would set that up tho.

Is there a way to see how many lanes are being used by a device?

[Bill_Gehrke]

rsukhram@tgioa.com wrote:

Is there a way to see how many lanes are being used by a device?

If you run GPU-Z you can see how many lanes each GPU is actually using when they are loaded.  You can actually run multiple instances of GPU-Z for each GPU card installed.  I also am having problems with my X99-E WS living up to its specifications even without a ThunderBolt card (three GPU's) and a SM951 M.2 SSD installed.

[Ren Sukhram]

gotcha Bill.  Im also a bit disappointed at the moment.  "Thoughtful" on the Asus forums mentioned something about other Asus boards having a TB reference :

I do not see where support for Thunderbolt has been added to the microcode for the ASUS X99-E WS motherboard.

Support for Thunderbolt was specifically revised and listed for the other ASUS X99 motherboards:

*X99-DELUXE; BIOS Version 1004; 10/23/2104

*X99-PRO; BIOS Version 1004; 10/23/2014

*X99-A; BIOS Version 1004; 10/23/2014

Hopefully, ASUS will add specific support for their ASUS ThunderBoltEX II card (or Thunderbot, in general) in a future BIOS update for their ASUS X99-E WS motherboard.

I wonder if there will be conflicts between the Thunderbolt controller, SATA Express controller, PCI Express with NVMe protocols, GPU, SATA controllers, etc., due to lack of available lanes. We need guidance in the form of a component matrix to help clients to better and successfully populate ASUS motherboards with old and newly emerging hardware.

My storage decision (the Cineraid) depends on the TB card which limits my GPU options + using the M.2 dive....SMH non-stop but trying to be positive and was thinking about offsetting all my rendering needs to my previous build.

I started a new thread here.  Let me know if you think this makes sense to do.

https://forums.adobe.com/thread/1821493

[ECBowen]

We ship the WS board with the TB cards. The latest bios works fine for recognizing the cards.

The cable I am referencing in the post above is the security header cable that connects the TB card to the board. That cable has to remain plugged in for the TB card to work and the settings in the bios to show up. They wont show if you disconnect that cable. So what you have to do is remove the card from the slot and lay it down on the bench with the system off but keep that cable plugged into the card and board. Then when you boot up and go into the bios the Thunderbolt controller options will be there under the advanced tab in advanced mode. AIC support is listed under those options.

The Resource charts build automatically by the board. You don't see or do anything. It just will restart shortly after you turn it on as it rebuilds those resources.

Yes the Samsung drive would use the Sata Express controller. You may not have to disable that to free enough resources for those cards. I did here because it was not being used.

Eric

ADK

[Ren Sukhram]

ok noted.  I am running a few VDP jobs at the moment so I cannot bring the system down but first chance I get I will try your settings.  Thanks for all the info!

[ECBowen]

There are some bios settings that have to be done to get more than 1 GPU to work with the TB card. Also you have to have the latest bios version which you do have. The TB card has to be in slot 2. You have to setup the bios correctly with just 1 GPU and then add the 2nd GPU once you have those settings in so the resource charts build around 2 cards. The 3rd card probably wont happen without an expansion chassis because of the slot spacing in most ATX cases.

Eric

ADK

[Ren Sukhram]

I currently have 1 GPU (Quadro 4000) in slot 1.  The TBolt card is in slot 2.

Would you happen to know what those settings should be?  My 2nd GPU will be the Tesla C2075 until i replace all my GPUS with the Titan-X.

Also note that I have a CM Cosmos II chasis so space is not an issue.

thanks ECB

[ECBowen]

I posted them in the red forums. I am pasting them below.

Try the following:

Pull TB card but leave cable plugged in.

Go into the bios

Go to advanced mode

Go to the boot tab

Disable fast boot

Int19 trap response - change to postponed

Above 4G Decoding - Change to enabled.

Save and exit.

Let the systemboard build the resource charts again which may take several reboots shortly after post. Then make sure it boots to the OS. Shut the system down and then put in the TB card. If it still doesnt work take the card out and disable AIC support totally. Then try it again. As long as you have the security cable plugged in it should still give you the TB card setting options. Also You may have to disable the Sata express port to free up resources. I did that here on a WS to get the latest bios to work with 2 980GTX cards and the Thunderbolt card with a Xeon chip

Eric

ADK

[Ren Sukhram]

Try the following:

Pull TB card but leave cable plugged in.

    you mean the 9-pin to 5-pin cable?

Let the systemboard build the resource charts again which may take several reboots shortly after post.

     is there something I should be looking at to confirm this?

Then make sure it boots to the OS. Shut the system down and then put in the TB card. If it still doesnt work take the card out and disable AIC support totally.

     where is AIC support?

Then try it again. As long as you have the security cable plugged in it should still give you the TB card setting options. Also You may have to disable the Sata express port to free up resources.

     Would this affect my M.2 XP941 drive...which happens to be my Main OS drive.

[Dustin_Farrell]

I worked on this for around 12 straight hours and finally found this post. Now I can sleep. EX3 card works with GPU expansion box now. Thank you! I owe you a beer if you are ever in Phoenix.

[dcwoodsdesign]

This is a known issue.  I have the same problem with 2 980 Ti's (yes brand spankin new).  I've tried 2 980's also and it seems to be a problem with the GPU UEFI based on dozens of other forum posts.  Funny thing is I've also tested with 2 Titan Blacks and the Titans work fine.  I was on the phone with both customer tech support from Asus and EVGA and both blamed it on each other.  After EXTENSIVE research it is a problem with both.  Asus BIOS hasn't dealt with modern GPU UEFI's (newer GPUs fail to show up on their "qualified vendor list" for their flagship X99 boards, and the GPU UEFI if disabled can lead to a successful TB2 card boot.  I've been struggling with this for months now but have not contacted EVGA to figure out what can be done with the GPU UEFI.  Ever get any more info than what I mentioned?  I would love to have closure on this!

[Ren Sukhram]

Sorry I can't give closure on this.  My config now has a single Titan-X with the TBolt card + M.2 for primary.  

[astross]

I have a single Titan-x with Thunderbolt card but I have had no luck getting to to run.  What is your slot configuration?

[Ren Sukhram]

I attached 2 pics for you to reference.  TBolt in Slot 2,  Titan-X in Slot 3.

[jamess55428008]

I'm having the same issues you are having.  Same MOBO, same ThunderboltEx II card and I'm running a single GTX 980 (just for the install, have a second one waiting to get hooked up). 

I'm losing my MIND! 

How did you end up finding these advanced menu settings in the BIOS?  I've followed everything listed above to no avail.  I see some of the menu options in BIOS but nothing that lists as Thunderbolt, nor the AIC options either...

For your hardware installation, did you have to connect the included displayport cable back to the Titan X in order for the Thunderbolt card to work?  Very confusing indeed considering we don't have any DisplayPort outputs on our specific MOBO model. 

To recap:

I first updated the BIOS (just released last month, July 2015)

Then I installed the driver from the included disc

Then powered down, installed the card in the second slot, and connected everything but the displayport cable.

Booted up and no sign of thunderbolt in the device manager, just a bunch of listings for "base system device" 

I've since tried the bottom most slot but no luck either. 

Please tell me exactly how you configured your system to work.  I'm gonna be losing sleep over this all weekend otherwise.

Thanks in advance!

[Ren Sukhram]

James,

I also had difficulty in the beginning and to be honest I am not sure if any of my BIOS versions opened up the TBolt menu but I do see one in my BIOS.  Also I use my TBolt card strictly to hook up an external Raid Storage device so I have not tried using the DisplayPort.  From a hardware side make you after you connect the TB card that you connect the ribbon.   I would advise that you use the drivers from the Asus site and not the disc since those are going to be more up to date.   Also if I am not mistaken the manual for the TB card says not to have the device installed when installing the drivers.

I took some pics of my BIOS settings that may be of some help.  Note on my 2nd pic along the tool bar there is an "Advanced" option.  In there is where I found "Intel(R) ThunderBolt"

[Ren Sukhram]

Also here is my device manager screenshot.  I also had a lot of unresolved base system drivers.  Install the Chipset driver again.  That worked for me.

[Ren Sukhram]

just curious if you are still experiencing this issue?  My system has been running (not to my total expectations) but running nonetheless so I have not tried updating any drivers or firmwares since.

Also do any of you have an issue where the NIC card does not connect to your domain unless your disable and enable?

[jasonwd]

I'm so glad you guys started talking about this...

I have an X99 Deluxe with a 5960K and a pair of 970s - built for 3D rendering grunt in February

When I built the system I messed with it no end and couldn't get the stupid thing to work, so it has been sitting there ever since, blocking air to the top GPU.

I just updated the BIOS and all the drivers for Win10 and, thanks to you guys' advice I feel metalled to have another go at making it work.

[dcwoodsdesign]

If you follow ECBowmen's advice, you'll get it to work.  I had almost the same setup and I got it to work.  I upgraded to an Asus X99 Sabertooth after a failure on the X99 Deluxe (thank god for Microcenter's warranty/return policy) and the same problem exists on that board.  Got it to work eventually on that too.  Best of luck!

[Ren Sukhram]

Anyone with this config install Windows 10 yet?  Im more concerned about the TBolt adapter since it connects to my main Raid 5 enclosure.

[Ren Sukhram]

FYI I just did a fresh install (via the UEFI method and a DVD) and used the latest drivers available for the TBoltII card and Areca Raid enclosure (which is only win 8.1) and got everything to work with no issues.

[Irenabys]

I have finally had success with the following configuration with Thunderbolt display routing and thunderbolt drive access in both Windows 10 and hackintosh OS X 10.10 (not no hot swapping in OS X)

ASUS X99-DELUXE BIOS Version 1901

Intel Core i7 5930K

EVGA GeForce GTX 970 4GB

ThunderboltEX II Card

Clover Bootloader

Windows 10 / Hackintosh OSX 10.10 Yosemite

In addition to the comments in this thread I found Josh Olsons description at the link below to be most useful and have added my install steps on stack exchange at:

bios - Thunderbolt on PC on an X99 motherboard (and the R9 295x2)? - Super User

Thanks to everyone who contributed system info here.