24 Replies Latest reply on Nov 3, 2010 8:51 PM by unique_screenname_here

    Heavy PixelBender calculations crash nvidia video driver

    Felix Dames Level 1

      When ever I do heavy calculations with Pixelbender mostly when doing long for loops the video driver crashes with the following error message:

       

      Error message is in german because of my german operation system

      Der Anzeigetreiber wurde nach einem Fehler wieder hergestellt

      Der Anzeigetreiber "NVIDIA Kernel Mode Driver, Version 258.69" reagiert nicht mehr und wurde wiederhergestellt.

       

      English translation of the error:

      The display driver has been restored after a failure

      The display driver "NVIDIA Kernel Mode Driver, Version 258.69" stops responding and has recovered.

       

      For testing and confirming this problem I created a simple Pixelbender Blur Filter with a for loop.

      With higher setting (BlurAmount) of this filter my video driver crashes!

       

      My system specs:

      Windows7 64bit

      Intel Core 2 Extreme CPU Q9300 @ 2.5ghz

      8GB Ram

      nVidia Quadro FX 2700M 512MB Driver Version 258.69

       

      On my older system (Windows Vista but same hardware) the error didn't show up! But I can not test this anymore because that system does not exist anymore...

       

      Maybe someone else could test this filter on there systems if this error is hardware, Windows7, nVidia driver or pixelbender related?

      The error only comes up in pixelbender all other software with heavy GPU usage (3ds Max, Games .... ) works fine on my system.

      When using the filter in After Effects it also crashes AE.

       

      I am working on some really nice filters but have not released them yet because of this instability problem.

       

      Any help or answer would be nice!

       

      Felix

       

      Here is the filter for testing:

       

      <languageVersion : 1.0;>

      kernel SimpleBlur
      <   namespace : "Felix Dames";
          vendor : "Felix Dames";
          version : 1;
          description : "Simple Blur";
      >
      {
          input image4 src;
          output pixel4 dst;

       

          parameter float bluramount
          <
          minValue: float(0.0);
          maxValue: float(1000.0);
          defaultValue: float(0.0);
          description: "Blur Amount";
          >;
         
          void evaluatePixel()
          {
              float2 pos = outCoord();
              pixel4 pixel;
             
              float pcounter = 0.0;
              for (float iy = -bluramount ; iy <= bluramount; iy += 1.0)
                  {
                  for (float ix = -bluramount; ix <= bluramount; ix += 1.0)
                      {
                      pixel = pixel + sampleNearest(src, pos + float2(ix,iy));
                      pcounter += 1.0;                          
                      }
                  }
                 
              pixel = pixel / pcounter;

              dst = pixel;
          }

      }

        • 1. Re: Heavy PixelBender calculations crash nvidia video driver
          unique_screenname_here Level 3

          I'll ask around about this but I don't believe there is much we can do about it. This timeout is under the control of the OS and I don't believe even the driver has any way to ignore it. Windows pretty much just pulls the rug out from underneath the driver and consequently us. There is a registry setting that can be used to modify the timeout but I believe this is a "debug-only" setting. Regardless, I'll ask around to see if there's something we can do (although I believe we would have done it had we known a way). Please feel free to post any relevant info you find on the topic--maybe we missed something. I believe you should be able to write a blur that meets your needs but does not put as much pressure on the driver.

          • 2. Re: Heavy PixelBender calculations crash nvidia video driver
            Royi A Level 1

            I had few problems with "Convolution" based filters as well.

            Most of them were gone when I defined "region" as documented.

             

            By the code you pasted I can't see that.

             

            The problem with releasing products based on Pixel Bender is there's no way to hide the code.

            I hope it will be fixed asap.

            • 3. Re: Heavy PixelBender calculations crash nvidia video driver
              Felix Dames Level 1

              Hey Royi A,

               

              I just cleared and simplified most of the code and also the region part of it so it is not too long for the post.

              The problem still exists with the region needed() and region changed() region code when implemented...

              Sorry that I cleared it out for simplicity

              But thanks for the hint

               

              I am now trying do come up with a better and faster box blur algorithm (horizontal & vertical)

              The biggest problem is that all is calculated in parallel and repeated for every pixel so it seems impossible to store allready sampled pixel data in an growing and shrinking array like most fastblur algorithm would do it.

               

              I allready came up with some solutions (faster rendertime and not with 2 neasted for loops) but they just do not look as good as my posted code example!

              They do not have the same smooth look.

               

              Maybe someone already have a solution for a fast and easy boxblur (Gaussian with weights is not needed) with variable bluramount/radius and would not mind to share some code?

               

              But back to the main topic of this post:

              Did you tried the example code and did your video/graphic driver crashes when turning the BlurAmount slider way up?

              Would be great to know if it is my system or just a normal behaviour of Windows shutting the driver down when the computation load is too high and takes to much time.

               

              Thanks so far

               

              Felix

              • 4. Re: Heavy PixelBender calculations crash nvidia video driver
                unique_screenname_here Level 3

                I tried it on my mac and it nearly hung the GPU (I stopped the processing). I believe it would have finished had I let it go but it's hard to say. I believe the same could be said for Windows. The problem is that the Windows OS is resetting the driver before processing can complete. I do not believe this is your machine; it is a Windows feature.

                 

                In any case, you'll see the identical problem with our basicBoxBlur sample if you increase the max blur radius. This is to be expected when using these "simple" box blur algorithms in Pixel Bender because as the blur radius increases so does the number of pixels you are sampling for each output pixel and, as you noted, there's no way to cache the results from previous computations. The best way to resolve this problem in Pixel Bender is to change how you blur. The box blur is separable so try doing the blur in two passes, one for horizontal and one for vertical. You could wire this up in a Pixel Bender graph if your target is AfterEffects or Photoshop and not Flash.

                 

                Out of curiosity, would it be useful to you (the community) if we provided optimized implementations of various "building block" filters, such as the Box and Gaussian blurs, that you could reference from your graphs or does that not fit your usage model(s)?

                • 5. Re: Heavy PixelBender calculations crash nvidia video driver
                  Royi A Level 1

                  I think it's easy to implement separable box blur.

                  That will give significant speed boost. Though a Blur Kernel of 20001 x 2001 is really too much :-).

                  Moreover you could easily move the Normalization Factor to the "evaluateDependents" section.

                   

                  Do you know how to implement separable Gaussian Blur?

                   

                  If you need me to write code, let me know.

                  • 6. Re: Heavy PixelBender calculations crash nvidia video driver
                    Felix Dames Level 1

                    Thanks for testing your result is what I thought about the problem... but what me really interested was if on a windows machine the driver would crash?

                    So a test with a Mac is not really representive but good to know that it seems OSX does not reset the driver when calculation takes too much time.

                    But for someone who would like to release a pixelbender filter it is important to know if a filter could crash the video driver and the corresponding application which it runs under. If that happens the application also crashes. Often it's not possible to save anymore. Pixelbender crashes completly and you have to kill the process in taskmanager. Then you lost the last lines of written code if not saved before...

                    Lets say on my fast system with a high end graphic board the filter runs fine but on a different low cost system with bad graphics board the filter crashes the video driver because operation took too long?!? that behavour is not acceptable for someone who develop. Don't get me wrong I know Pixelbender is not yet finished but this is an aspect to think about. As a developer you want to garantee stability.

                     

                    I know that it would be possible to separately blur in horizontal and vertical pass but then you need to go with graphs.

                    If you want to have the blur directly in the kernel maybe as a function to make function calls as I would prefer as far as I know it's not possible to get the same blur result because for the second pass you need the first pass to be finished before. Or are I'm wrong or too stupid?

                    Maybe I have to take a deeper look into graphs up to now I didn't used that feature.

                     

                    Generally some "building blocks" are always nice too look at so feel free to share them with the community. I would be interested in all kinds of blurs/convolve things where more than one source pixel is involved.

                    I think espacially for pixel bender it is a good decision to provide some more examples because pixel bender programming is sometimes totally diffrent compared to other languages (calculation in parallel and repetitive for every pixel....) this make some things super easy (calculations where just one pixel is involved) and other parts of programming totaly difficult to develop. Maybe it would be good if pixelbender could do both programming models. just as a feature request.

                    1. model as it it now parallel and repetitive for every pixel

                    2. model more like other languages. calculation for a whole frame pixel by pixel so caching would be possible

                    • 7. Re: Heavy PixelBender calculations crash nvidia video driver
                      unique_screenname_here Level 3

                      The test on the mac showed me that your filter is very compute-intensive at large radii. I used that data, combined the message in your post, and some experience to come to the conclusion that windows was resetting the driver because the filter took to long to complete. It's certainly possible that the driver crashed of its own accord, but I don't believe that's the case here. Driver crashes are sadly not out of the norm on Windows. Consider yourself a lucky man if this is the first reset/crash you've seen. I believe you're at least three driver releases out-of-date so the first thing you should do is update your driver.

                      • 8. Re: Heavy PixelBender calculations crash nvidia video driver
                        unique_screenname_here Level 3

                        FYI: evaluateDependents is a per-frame operation so you can do calculations in that function that will happen only once per-frame. That said, there are a limited number of operations that can be performed in evaluateDependents (for instance sample functions cannot be called from evaluateDependents).

                        • 9. Re: Heavy PixelBender calculations crash nvidia video driver
                          Felix Dames Level 1

                          I know nobody needs to blur by a radius of 1000 pixel but up to 200-300 pixels is not uncommon in postproduction.

                          My graphic driver crashes at nearly 100 pixel radius!

                          My SimpleBlur kernel was made for testing if used on a windows system the graphic driver would crash that's why I set the maximum bluramount slider to 1000.

                           

                          If you could share a faster version of my SimpleBlur kernel which produce the same results, operate in one kernel (no graphs involved) with variable bluramount/radius that would be great and would help me a lot!

                           

                          Sometimes it really helps to have a working code example to lern from.

                           

                          So thanks if you do it but I can't pay you for it. sorry

                          • 10. Re: Heavy PixelBender calculations crash nvidia video driver
                            unique_screenname_here Level 3

                            You don't necessarily have to use a graph. You can run a horizontal pass over the image and then use the output of that pass as the input to a vertical pass. Graphs just simplify the process described in the previous sentence and act as a convenient package for kernels.

                            • 11. Re: Heavy PixelBender calculations crash nvidia video driver
                              Royi A Level 1

                              How could you use the result of the first pass for the second pass without graphs?

                              You need the whole Horizontal Blurred image for the second pass.

                               

                              As far as the documentation goes, the only way to do so is using Graphs.

                               

                              Could you make us smarter?

                              • 12. Re: Heavy PixelBender calculations crash nvidia video driver
                                Felix Dames Level 1

                                Hey sorry but this is the only one driver crash (and the first ever) I got on my system and it only happens when using pixelbender!

                                 

                                Even when doing heavy GPU rendering with for example Mental Images IRAY renderer which is a gpu renderer and uses the gpu and the cpu at the same time by 100% for hours when rendering I do not get any crashes!

                                Then my mouse starts moving slowly and it slows down all operating system graphics because my gpu is so heavy loaded with calculations but nothing crashes!

                                 

                                But Pixelbender !?!

                                 

                                My driver version is nearly a month old. It happend with every driver I testet in the past! (about 5-6 different versions on win7)

                                All I wanted to know is if someone else get this driver crash on a windows system! Espacially windows 7 64bit.

                                 

                                Maybe I test the newest nvidia driver but I don't think it will change anything.

                                • 13. Re: Heavy PixelBender calculations crash nvidia video driver
                                  unique_screenname_here Level 3

                                  Blur an image. Save the result. Blur the result. Sure, it's not ideal but it is effectively the "poor man's" graph. That was the way things worked before we introduced Pixel Bender graphs. It's hard to say what workflows will and won't work for people given so little information. For instance, it's not clear to me why Felix doesn't want to use graphs; maybe there's some restriction in his workflow. In any case, the procedure I listed above could be easily automated in Photoshop and there you go, graphs without graphs.

                                  • 14. Re: Heavy PixelBender calculations crash nvidia video driver
                                    Felix Dames Level 1

                                    I think Royi A is absolute right!

                                    When it's possible to do it without graphs please show me!

                                    • 15. Re: Heavy PixelBender calculations crash nvidia video driver
                                      unique_screenname_here Level 3

                                      Are you telling me the same blur algorithm works for the same radii on the same image in Mental Ray? If so, then maybe there is something we're doing wrong on our side. I don't believe any crashes are ever okay but sadly I have found them to be the norm when working with the GPU on windows. Regardless, it is the driver and not Pixel Bender that is crashing or being reset and it's largely out of our control. There are some things, such as tiling, that we would like to do in upcoming releases of Pixel Bender which would hopefully minimize the likelihood of driver crashes/resets. I can try to verify the crash/reset on windows if it would make you more comfortable, but I expect that I will see exactly what you are seeing.

                                      • 16. Re: Heavy PixelBender calculations crash nvidia video driver
                                        unique_screenname_here Level 3

                                        Blur an image. Save the result. Blur the result. Sure, it's not ideal but it is effectively the "poor man's" graph. The was the way things worked before we introduced Pixel Bender graphs. It's hard to say what workflows will and won't work for people given so little information. For instance, it's not clear to me why Felix doesn't want to use graphs; maybe there's some restriction in his workflow. In any case, the procedure I listed above could be easily automated in Photoshop and there you go, graphs without graphs.

                                        • 17. Re: Heavy PixelBender calculations crash nvidia video driver
                                          Felix Dames Level 1

                                          Are you telling me the same blur algorithm works for the same radii on the same image in Mental Ray?

                                          No!

                                           

                                          I talked about Mental Images IRAY (not mental ray this is cpu only ) IRAY is a GPU & CPU renderer.

                                           

                                          I just wanted to point out that there are some GPU rendering applications out there that do extremly heavy calculations on the gpu for over hours of time without crashing any driver!

                                          • 18. Re: Heavy PixelBender calculations crash nvidia video driver
                                            unique_screenname_here Level 3

                                            Do you get a crash when you run this Pixel Bender filter on the CPU?

                                            • 19. Re: Heavy PixelBender calculations crash nvidia video driver
                                              Felix Dames Level 1

                                              Ok my filter works seperatly on the three RGB channels and different calculated results of these channels get blured differently and depending on other calculations and with different blur algorithms. I have put all the blur algorithms (zoom blur, only horizontal or vertical blur, blurs which depends on positions like motion blurs...) in functions for better looking code and easyness so I can make function calls whenever needed. I think it would not work with graphs! That's why I want it directly in a kernel. All these blurs work fine and fast because they are 1 dimention/vector but a horizontal & vertical blur is 2 dimention/vector.

                                               

                                              The filter is for After Effects only.

                                              The filter is doing lens distortion with chromatic aberration and other funny things... but more I'm not willing to talk about. sorry

                                              Here you have a teaser of the filter:

                                              rgb.jpg

                                               

                                              And the evaluateDependents() thing does not help because as you already pointed out sampleNearest() function inside this code block does not work or is not allowed!

                                               

                                              I already did a separation for the horizontal and vertical pass with two for loops in series and in one kernel! but it does not look the same because you need the entire image result of the first pass for the second pass to get the same result. That's only possible with graphs I think!

                                               

                                              Inside a kernel you only have access to the result of one pixel not the whole frame/image result!

                                               

                                              I already came up with a what i call StarBlur kernel which only have one for loop and which is sampling the pixels in a star like pattern which is much faster and give ok results but not that smooth like the sample I provided in the original post.

                                              • 20. Re: Heavy PixelBender calculations crash nvidia video driver
                                                Felix Dames Level 1

                                                When run in CPU mode it does not crash! How could it? The GPU is not involved so the GPU driver can't crash

                                                • 21. Re: Heavy PixelBender calculations crash nvidia video driver
                                                  unique_screenname_here Level 3

                                                  Right, that was kind of my point. There is your solution--run it on the CPU. AfterEffects does not process end-user Pixel Bender filters on the GPU so your final product will run on the CPU anyway. You should be able to use a very large radius in your original blur on the CPU and everything will work provided that you have the memory and processor cycles.

                                                  • 22. Re: Heavy PixelBender calculations crash nvidia video driver
                                                    unique_screenname_here Level 3

                                                    Right, you only have access to one output pixel but that does not render evaluateDependents unusable. For instance, you could use it to create a LUT of weights that you use as part of your blur thus making the blur indepent of the radius size. Something kind of like this:

                                                     

                                                     

                                                    <languageVersion: 1.0;>

                                                    kernel poisson_disc_blur

                                                    <

                                                    namespace: "";

                                                       vendor: "";

                                                      version: 0;

                                                     

                                                    documentation: "developer.amd.com/gpu_assets/GDC06-ATI_Session-Oat-ShaderTricks.pdf";

                                                    >

                                                    {

                                                    parameter float radius

                                                    <

                                                    defaultValue:   3.0;

                                                        minValue:   0.0;

                                                        maxValue: 1000.0;

                                                    >;

                                                     

                                                    dependent float samples[24];

                                                     

                                                    input  image4 src;

                                                    output pixel4 dst;

                                                     

                                                    void evaluateDependents()

                                                    {

                                                    samples[0]  = -0.326212;

                                                    samples[1]  = -0.40581;

                                                    samples[2]  = -0.840144;

                                                    samples[3]  = -0.07358;

                                                    samples[4]  = -0.695914;

                                                    samples[5]  =  0.457137;

                                                    samples[6]  = -0.203345;

                                                    samples[7]  =  0.620716;

                                                    samples[8]  =  0.96234;

                                                    samples[9]  = -0.194983;

                                                    samples[10] =  0.473434;

                                                    samples[11] = -0.480026;

                                                    samples[12] =  0.519456;

                                                    samples[13] =  0.767022;

                                                    samples[14] =  0.185461;

                                                    samples[15] = -0.893124;

                                                    samples[16] =  0.507431;

                                                    samples[17] =  0.064425;

                                                    samples[18] =  0.89642;

                                                    samples[19] =  0.412458;

                                                    samples[20] = -0.32194;

                                                    samples[21] = -0.932615;

                                                    samples[22] = -0.791559;

                                                    samples[23] = -0.59771;

                                                    }

                                                     

                                                    void evaluatePixel()

                                                    {

                                                    float denominator = 13.0;

                                                    float2 coord = outCoord();

                                                    pixel4 accumulator = sampleNearest(src, coord);

                                                     

                                                    for (int i = 0; i < 24; i += 2)

                                                    {

                                                    float2 sampleCoord = coord + (float2(samples[i], samples[i+1]) * radius);

                                                    accumulator.rgb += sampleNearest(src, sampleCoord).rgb;

                                                    }

                                                    accumulator.rgb /= denominator;

                                                    dst = accumulator;

                                                    }

                                                    }

                                                     

                                                     

                                                    Anyway, this isn't a great example but trust me--there's lots you can do with evaluateDependents so don't sell it, or yourself, short. I would be happy to help you build and wire up the graph if you would like. Let me know.

                                                    • 23. Re: Heavy PixelBender calculations crash nvidia video driver
                                                      Felix Dames Level 1

                                                      Ahh no that's not my solution!

                                                      When run in cpu mode it takes ages to render!

                                                      Does After Effects CS5 still run them in CPU mode??? I thought that changed from CS4 to CS5?

                                                      And by the way if you are right in my opinion it's a bad decision that third party pixelbender filters only run in cpu mode in After Effects!!!

                                                      That totally makes no sense to build filters which benefit from the parallel processing power of gpu and then cripple them down to run on the cpu!

                                                      Next thing I would like to see is a plugin based type for pixelbender in After Effects like it is for Photoshop so that it is possible to update Pixelbender in After Effects and you don't have to wait for the next release of After Effects to get the latest pixelbender support!

                                                      • 24. Re: Heavy PixelBender calculations crash nvidia video driver
                                                        unique_screenname_here Level 3

                                                        Oh, my mistake. I thought you said you let Mental run for hours so I just assumed you wouldn't mind letting Pixel Bender run for minutes on the CPU. Please do give it a try in AE--the lack of GPU support for Pixel Bender filters does not mean a lack of parallelism as we will process on available CPUs and CPU cores. Pixel Bender filters are not GPU accelerated in AE CS5. I wish they were. Make sure to let the AE team know that this is a feature you want. They are a great team and really appreciate customer feedback.