Kernel Crash during render with CUDA and Nvidia Quadro 4000 - total frustration!

[LindyHop1943]

I just want to throw the computer and Premiere out of the window. I'm so frustrated that I want to stop doing video work for good! Once a year I have to edit large instructional DVDs for a dance camp. This year I'm having major issues that kill my life and my health. I have spent more than 3 weeks on that issue without any result.

Problem: When I export a section of the Premiere timeline the kernel graphics card driver crashes. About 1 to 10 minutes into exports (direct export, no queued AME) the screen goes black and Windows reports a NVidia driver kernel error. The driver is recovered and Premiere hangs. This only happens on CUDA export not with the software option of Mercury. The error is not related to a certain clip or portion of it. If it crashes on a certain clip and I restart Premiere this clip will be rendered fine the next time. The crashes happen always after a certain but varying time somewhere during the rendering.

Since the source material is 2K I need to scale it for DVD output. Therefore I need the CUDA support for the output quality. Premiere uses the better Lancos scaling algorithm only with CUDA support and when exported directly (not via AME—at least not with CC2014). Without the CUDA support the scaled down pictures looks not sharp enough. DVD already has a low resolution. Unfortunatey Premiere has no option of sharpening the output.

For troubleshooting I have tried:
- Premiere Pro 7, 7.2, 8.0, 8.0.1 and the actual 8.2
- various nVidia drivers from one year ago to the actual one
- different user account
- different render formats (for the project I need to output to MPEG-DVD)
- newly created projects and timelines
- checked the temperature with GPU-Z
- exported to external and internal drives

- source files form external and internal drives

- the latest step was a complete reinstall of the system (Win7, updates, CC) incl. BIOS update

Nothing stopped the kernel crashes when using the CUDA.

I contacted PNY and NVIDIA. I run stress tests and other tests on the card. It shows no problem at all. The graphics driver doesn't crash on any other usage.The card does not show any issues.

I have a ASUS P6T board with i7 920, 12 GB RAM, NVIDIA  PNY Quadro 4000. Win 7 Ultimate SP1, all updates and patches installed. The source material of this year is from a Sony F7 in 2K.

It looks to my very much as if Premiere has a bug that causes the driver to crash. It worked fine until last year's project. The only changes are the constant OS and software updates. No hardware has been changed.

The randomness of the crashes would normally suspect thermal issues, but since the computer doesn't show any thermal issues during other work or stress tests and only with CUDA I rather suspect an issues of the parallel programming within Premiere when addressing the CUDA.

Does anyone have another clue?

[RoninEdits]

other possibilities to test, does this happen only with the sony footage or other media? do you have any extra hardware like aja card or third party plugins that premiere uses? can also troubleshoot the timeline to see if its a certain premiere effect or nesting thats causing it.

can look at this, if this matches your error.

https://forums.geforce.com/default/topic/413110/geforce-drivers/the-nvlddmkm-error-what-is-it-an-fyi...

prime95 and memtest86 are also good programs to stress test the computer, to see if there are any hardware problems.

and another gpu tester  Video memory stress test

[LindyHop1943]

Thanks for your reply and the tips. The link looks promising. I will work on it next.

I tried different footage and the problem happened, too. There is no extra hardware and not third party plug-ins. The timeline only has a fade in and out on each clip plus a title and an overlay video at the begining (animated lower third) of each section. As I wrote the error occurs randomly and when I render the same clip again it normally works on the second try, very seldom I needed a third try.

[LindyHop1943]

I checked the link and tried a test with higher setting of the regsitry keys. It still made no difference. my original settings were already higher than the MS default values. I set them back to their original value.

I checked the temperature and fans of the cards together with NVIDA and PNY support personal. The card runs fully within the specs and creates no crashes except during rendering from Premiere. I run a memory test of the computer, too. No issues.

So I guess it leaves me to dump the computer or Premiere.

[JEShort01]

I'd give a better than 75% chance that you have a problem with either your power supply or video card.

Since the video card is easy to change, try to buy a GTX 970 video card locally (with 30 day return policy) and swap out your Quadro card. You will want to delete the Quadro driver, etc. before turning the PC off for the board swap.

If that fixes your problem, then that card will provide much faster DVD exports that your current card will too.

If that does not fix your problem, or makes your crashes even more frequent (I would guess that the current draw on a 970 is probably higher than the Quadro), then change out your power supply.

Additional comments:

- traditional diagnostics are really pretty lousy at pushing hardware as hard as Adobe software can

- I've seen exactly the symptoms that you are having from a weak power supply and from a bad video card on separate occasions using Premiere

Good luck!!!

Jim

[LindyHop1943]

Hi Jim,

thanks for your input. Changing the card is my next option if win 8.1 does not get rid of the issue (the system is still installing).

In Germany you normally can't return a card if you opened it. I would need to order it online to have 14 days for return. The card and the power supply worked fine for many years. Yes, they could go bad, but since none of the tests showed any issues and I have no problems anywhere else even with CUDA my bet is still on Premiere. The issues started with CC (v7). Since the newer camera material is not supported on earlier versions I could not go back further. Looking at the unbelievable list of issues that Adobe software has since they moved to CC (and I eperienced many of them) plus everything I tested so far makes it much more likely that the issue is with Premiere.

Some years back I had similar problems with Matrox and AXIO. It took me 4 weeks of tests to finally prove that it was a bug in the software that made Premiere crash. Then they finally found and admited it. It just cost me one month of my time unpaid! So I got rid of it when Premiere supported CUDA.

I will run my last test with Win 8.1 and will try a new card. If this doesn't work I will dump CC and video editing.

[Bill_Gehrke]

I just checked the power for the 4000 and a GTX 970 , almost the same at 142 and 145 watts.  I would suggest that you get a GTX 960 which is only 120 watts and costs much less than the GTX 970 but the GTX 960 has 4x as many CUDA cores and slightly higher memory bandwidth than the Quadro AND both the Quadro 4000 and GTX 960 only require one 6-pin power connector!  The GTX 970 requires two 6-pin power connectors.

[LindyHop1943]

Hi Bill, thank you very much for checking this out and the thorough details.

I'm leaving for a week on Saturday, so I might not be able to test everything tomorrow. I also need to check if I can return the card when I buy it in a store. I will post my results when I'm back and have made the tests.

Thanks, Marcus

[LindyHop1943]

Hi Bill, just another question. Would the 980 or 980 ti be worth the extra money? If I don't throw the computer and Premiere out of the window I will need to work on 4K footage. I might update the board and processor next year.

[RoninEdits]

do you actually know which power supply you have in the computer? model number and watts? your cpu is rated 130w and video card at 142w, so a good power supply rated above 500w should be able to keep up. if the power supply is of low quality, and close to the watts needed when system is under full load, its performance may have been reduced over the years, and is no longer able to keep up to the demands of the system.

for your i7-920, anything above the gtx 960 (120w), or possibly even the gtx 750 ti (60w, some models need no power connector, and 50% faster than the quadro), may be overkill. when you upgrade the system, then the gtx 980 ti would be better for 4k. asus and gigabyte have some of the highest reliability for all the various pc parts they make.

to eric's point of it being a premiere bug, have you tried installing cc2015 to see if adobe may have improved/fixed something? you can install it along side the other premiere versions, just choose advanced and tell the installer not to remove the previous version.

[LindyHop1943]

Hi Ronin,

I can't read the full sign on the power supply. I need to unsrew it, but the machine is updating at the moment. It looks like it is a 700 W supply. I remember that I upgraded the initial supply when I switched to the Quadro. I also had an Axio card in the system and I have 6 hard drives. The Axio is gone

I upgraded to CC2015 and teh issue is still there.

[ECBowen]

You have some variables at play here that are triggering this with the GPU acceleration. It could be the media, FX, driver version and install method, or something else. I would need to remote in and see what all is involved here because there are to many possibilities. If you test the video card via 3D Mark and it's fine then it has to be 1 of those variables. I suspect the card is fine especially since you contacted Nvidia. PNY's support may not be the best but Nvidia support knows what they are doing.

Eric

ADK

[LindyHop1943]

Bill, is their any manufacture that you would prefer or recommend. The versions seem to differ not only in reliability and drivers.

I have two free 6-pin connectors. Would you still recommend the 970?

Thanks Marcus

[LindyHop1943]

‌Hi Ronin,

I just read that I had to reboot the system after I changed the registry keys. I did not do that. I will give it another try when I'm back In a week.

My dealer also said he will provide me a new graphic card to test. If it works I will keep it if not I can give it back. So I have to decide which one I will take.

I came across the links you send me in your post before. Most people had frequent issues. Since I only have issues when using the CUDA in Premiere on render it might be a different source.

It was my understanding so far that Premiere uses the CUDA for previews and for scaling, but not for export rendering, like h.264 or MPEG. I read now that the CUDA on the older chipsets is used, but not on Kepler and Maxwell, due to a change in Nvidia SDK that Premier doesnot support, yet. This would mean that my old Quadro would speed up the render to MPEG-DVD, but the newer 900 cards would not. Is that true? I read that Premiere 2015 was supposed to support the new SDK, but still isn't. Is that true?

Thanks for all of your help!

Marcus

[Bill_Gehrke]

LindyHop1943 wrote:

It was my understanding so far that Premiere uses the CUDA for previews and for scaling, but not for export rendering, like h.264 or MPEG. I read now that the CUDA on the older chipsets is used, but not on Kepler and Maxwell, due to a change in Nvidia SDK that Premier doesnot support, yet. This would mean that my old Quadro would speed up the render to MPEG-DVD, but the newer 900 cards would not. Is that true? I read that Premiere 2015 was supposed to support the new SDK, but still isn't. Is that true?

Thanks for all of your help!

Marcus

I do not know where that information comes from but it is wrong.  With our Premiere Pro BenchMark you can see major improvements when exporting MPEG2-DVD and it is not all do to scaling.  Also if I export the H.264 (HD to HD--no scaling involved) timeline with GPU acceleration on this laptop it takes 196 seconds, without GPU acceleration it requires 1380 seconds.  It is very dependent of the media, features, and effects that you use.  There have been continuous PPBM benchmark results improvements as nVidia has released new cards up until they released the Maxwell cards but the Maxwell cards are roughly equal to the previous cards but have two major other improvements.

1. They use much less power.
2. They offer HDMI 2.0 (4K resolution) 

[LindyHop1943]

Hi Bill, hi Ronin, thanks for your input.

That at least older versions of Premiere did not use CUDA for rendering was a stement I got confirmed from Adobe employees a couples of years ago. I first found it in a blog entry of the Premiere Pro blog:

http://blogs.adobe.com/premierepro/2010/10/scaling-in-premiere-pro-cs5.html

http://blogs.adobe.com/premierepro/2011/02/cuda-mercury-playback-engine-and-adobe-premiere-pro.html

In the second blog entry it is stated that Premiere dos NOT use CUDA/Open CL for encoding and decoding.

There is also talk about it at NVidia GPU-accelerated H264-encoder plugin, ready for public testing

and CreativeCOW.

What I don't know is if anything has changed with CC 2015. Does anyone now the link to an updated list of what is CUDA supported in Premiere toda?.

Marcus

[RoninEdits]

i think its safe to say premiere still doesn't use the gpu to render, thats why they made that nvenc plugin. nvidia has hardware decoding for some formats, like mp4. i would hope premiere is using cuda for decoding, but not sure that it is. gpu-z and windows task manager should show the offload of the decode onto the gpu if it does.

for things that have changed in cc2015, i think adobe has only added cuda use for new features, like morph cut and the lumetri color panel.

Adobe Premiere Pro CC - Faster Video Editing | NVIDIA

[ECBowen]

You are confusing Rendering with decoding and encoding. GPU acceleration in Premiere is used for Realtime playback and Rendering for the same aspects of the frame. The GPU acceleration handles scaling, interpolation, and many FX. The Decoding and encoding is cpu processed. This is regardless of whether it's playback or render ie export. This has not changed since CS5. The differences with CC, CC 2014, and CC2015 have been additions to the acceleration. GPU accelerated Debayering and more FX have been added to the GPU accelerated functions. However the Decoding and Encoding is still CPU based and has not changed. The 900 series cards work for the GPU acceleration the same as the 700 series cards. The only compatibility issue with the 900 series cards was Ray Tracer support in AE. However Ray Tracer is legacy now in favor of the link to C4D. So the support is not needed anymore since Adobe is done developing it.

Eric

ADK

[LindyHop1943]

Hi Eric, you are right. I know it and just saw that I wrote it wrongly in my post. It was meant to be different.

In the meanwhile I'm back home and got a new card (Inno3D 980 ti). I took the high-end card, because I wanted to upgrade the system soon for 4k work. Anyway despite the new graphics card I still have the same problems. During the render I'm getting the driver kernel crashes.

I tried the registry change that I tried before, but didn't reboot the system. Strangly the registry had no settings for Tdr. When the quadro was installed the keys were there. Anyway I added a TdrLevel key and set it to 0 to turn of Tdr. I rebooted the system. No change.

I didn't remember if I delted the media chache so I did this, too. Still crashed.

I used msconfig to turn of the AV services (Gdata) and I also removed all the fonts that were not system fonts or the two fonts I used in the titles. Still crashed.

My last two trys will be:

1.) Open the file on my notebook (it has a Nvdia card) and see if it crashes there too. I just need to upgrade to CC2015 there.

2.) Turn off the title track and the track with the lower third and vice versa to see if the source video files are the sole cause.

I strongly believe in the meanwhile that it is a premiere bug with the handling or processing of the file format (2k Sony FS7, MXF, AVC)

I'm still extremely frustrated for all that wasted time. As I wrote before I had a smilar problem with Matrox Axio. After 4 weeks of wasted time and money I was able to prove that it was the Axio softwrae/encoding tthat cause the error 0 issues. After that I sent the the files and Matrox confirmed the bug (it was programming bug handling spanned and striped clips).

Marcus

P.S.: Is there someone of the Adobe support staff reading this and able to bring this to the attention of the right people?

[LindyHop1943]

I connected the extrernal HD to my notebook (Sony, i7 2670QM, 2.2GHz, GT540M, 8GB). I rendered a 30 minute clip without problems. I then connected the HD back to the desktop computer but used a different cable and a different USB port (USB 2.0). I was able to render the same 30 minute clip without a crash on the desktop, too. On both computers the render took about 18 minutes. I had instances where the render was longer than 20 minutes before the driver crashed. I need to go home. It is 12 midnight, so I have not enough time anymore for a longer test, but I will keep the desktop rendering over night. Tomorrow I will know if it crashed again or went through. It is just stange, because I copied the files to an internal drive before to make sure that the drive is not the issue. The external drive was still connected, but it wasn't used.

I also experienced another strange behaviour with the drive. I changed the drive letter, which resulted in a hang of the disc manager, the explorer and IE. Killing the process and restarting it still didn't help. I had to rebot the system. This happened on both the desktop and the notebook. Either it is an issue with the drive (I used a different cable on the notebook) or one of the Windows patches (Win7 and Win 8.1) or the AV causes that issue. I will report more tomorrow.

Marcus

[RoninEdits]

i wouldn't think the external hdd would cause the graphics crash, but stranger things have happened. windows will halt everything until it can re-establish or disconnect the drive. so even if the media was on an internal drive, if any program or even windows was looking at a file on the external hdd and it stopped responding for a moment, windows would pause and the chain reaction would begin.

adobe staff do sometimes read and chime in. Kevin-Monahan‌  may be able to pass it along if the problem is found, also there is a bug report form. Adobe - Feature Request/Bug Report Form

[ECBowen]

Memory management with GPU acceleration is critical to the entire process. If the drive latency takes to long to provide the data then the latency will exceed what the player expects with the GPU acceleration and the Nvidia driver will crash because the data required to finish the processing of the current threads does not arrive in system ram in time. This error shows as an Nvidia driver error since that is where the error reference occurred but the cause is the latency of the data pipeline. That is one of the main reasons the variables with this are so many and why I asked to remote in. Memory management can be effected by many variables and often times you have to look at all of them before you spot the problem.

Eric

ADK

[LindyHop1943]

I really don't get it. The render worked fine on the desktop. All clips were rendered and there was no crash when I came back in the office today.

What was different now? It was a different cable, a diffferent USB port and I copied the project file with explorer before I opened it on the notebook and saved it there. So to double check I plugged the HD with the same old cable in the same old USB port and opened the same old file. So the error should come up agan, but no!?!?!

The only other difference I could think off was the network cable that I unplugged due to the turned off AV. So I run another test with the plugged in cable. It worked again! I'm completely puzzled. For three weeks I tried everything and despite a re-installation it did not work and now it works???? There must be something else that I miss, but I can't think of it.

I will turn on the AV again and install the removed fonts again. I can also remove the registry setting and then see if the error comes back. There must be something that caused the error. I still can't see it.

[LindyHop1943]

So I reset the registry entry, enabled all services again and the render still works. I'm a living questionmark of the size of the Marshmallow man of Ghostbusters.

[LindyHop1943]

Now I also installed all previous fonts again. Everything is like yesteday morning, but now the render still works without any trouble! (???)