1 Reply Latest reply on Mar 24, 2010 4:05 PM by Bartosz Zaczyński

    Faulty calculation

    Bartosz Zaczyński

      I'm using PB kernel (see below) in an AIR / Flex application to speed up some calculations. The data I supply is a Vector.<Number> of floating point precision pixels. Every pixel is stored as three consecutive Numbers respectively for R,G,B channels, which corresponds to image3 and pixel3 data types (well, if you forget the conversion between IEEE 754 double and float). I only need to store the result of my calculation on G-channel which I chose for luminance. However, only the first four pixels are computed correctly. Starting from the 5th one there seem to be a problem with incorrect offsets of the channels, I suppose. To make matters worse, after removing the last division "... / div" from my kernel this partial result is fine. Identical code in AS3 works as expected. Here's a sample output of both methods:

       

      ActionScript 3Pixel Bender
      0.31890034675598145
      0.8537830501553823
      0.32749107480049133
      0.31890034675598145
      0.853783130645752
      0.32749107480049133
      0.3187195956707001
      0.8779068260949753
      0.3277072608470917
      0.3187195956707001
      0.8779069185256958
      0.3277072608470917
      0.3195112645626068
      0.8675161312801771
      0.32622960209846497
      0.3195112645626068
      0.8675161600112915
      0.32622960209846497
      0.31998929381370544
      0.8910256334295363
      0.3283438980579376
      0.31998929381370544
      0.8910257816314697
      0.3283438980579376
      0.32212474942207336
      0.8751980313098534
      0.3289455771446228
      2.47822642326355
      0.5179606080055237
      2.1203341484069824
      0.32434457540512085
      0.8869450816322106
      0.3317764401435852
      2.673691511154175
      0.5195903778076172
      2.147810220718384
      0.32753363251686096
      0.8913912862998146
      0.3324755132198334
      2.75229811668396
      0.5199907422065735
      2.1358659267425537
      0.3291414678096771
      0.8930972521677654
      0.33320507407188416
      2.7831685543060303
      0.5204076170921326
      2.1631364822387695
      0.3270454406738281
      0.9043374247870719
      0.33148670196533203
      2.9969263076782227
      0.5194242596626282
      1.8038333654403687
      0.3232068419456482
      0.9140460786037383
      0.3305985629558563
      3.197050094604492
      0.5189139246940613
      1.8051812648773193

       

      The kernel:

       

      <languageVersion : 1.0;>
      kernel Foobar
      <   namespace : "Your Namespace";
          vendor : "Your Vendor";
          version : 1;
          description : "your description";
      >
      {
          input image3 src;
          output pixel3 dst;

       

          parameter float avgY;
          parameter float maxY;
          parameter float exp;
          parameter float div;

       

          void evaluatePixel() {

              pixel3 p = sampleNearest(src,outCoord());


              float Y = p.g / avgY;
              p.g = (log(Y + 1.0) / log(2.0 + pow(Y / maxY, exp) * 8.0)) / div;


              dst = p;
          }
      }

       

       

      Equivalent ActionScript 3 code:

       

      for (i = 1; i < data.length; i += 3) {
         var Y:Number = data[i] / avgY;
         data[i] = (Math.log(Y + 1) / Math.log(2 + Math.pow(Y / maxY, exp) * 8)) / div;
      }

       

        • 1. Re: Faulty calculation
          Bartosz Zaczyński Level 1

          Further analysis revealed classic symptoms of the buffer overflow error. First of all, using a literal value rather than parameter for avgY yields correct results. Adding a dummy variable results in a shift of the intermediate values. Declaring yet more variables leads to overwriting one of the channels with the given dummy value (example below).

           

          At first I thought it might be due to erroneous PB compiler so I disassembled the kernel code using haXe (http://ncannasse.fr/projects/pbj) and Tinic's Uro C++ tool (http://www.kaourantin.net/2008/09/pixel-bender-pbj-files.html). However, I carefully scrutinized the assembly code and have to admit it's perfectly fine. Perhaps the problem lies with the implementation of Shader class within Flash? Or maybe I'm utterly wrong?

           

          Desperately looking for a solution.

           

           

          Example:

           

          p.g = p.g / avgY;2 dummy float variables4 dummy float variables
          0.31890034675598145
          35.5465202331543
          0.32749107480049133
          0.31890034675598145
          0.853783130645752
          0.32749107480049133
          0.31890034675598145
          0.853783130645752
          0.32749107480049133
          0.3187195956707001
          41.44520950317383
          0.3277072608470917
          0.3187195956707001
          0.8779069185256958
          0.3277072608470917
          0.3187195956707001
          0.8779069185256958
          0.3277072608470917
          0.3195112645626068
          38.77558898925781
          0.32622960209846497
          0.3195112645626068
          0.8675161600112915
          0.32622960209846497
          0.3195112645626068
          0.8675161600112915
          0.32622960209846497
          0.31998929381370544
          45.1257209777832
          0.3283438980579376
          0.31998929381370544
          0.8910257816314697
          0.3283438980579376
          0.31998929381370544
          0.8910257816314697
          0.3283438980579376
          0.32212474942207336
          40.729286193847656
          0.3289455771446228
          2.47822642326355
          0.5179606080055237
          35.5465202331543
          2.47822642326355
          0.5179606080055237
          2.200000047683716   // one of the variables was 2.2
          0.32434457540512085
          43.94172668457031
          0.3317764401435852
          2.673691511154175
          0.5195903778076172
          41.44520950317383
          2.673691511154175
          0.5195903778076172
          2.200000047683716
          0.32753363251686096
          45.233612060546875
          0.3324755132198334
          2.75229811668396
          0.5199907422065735
          38.77558898925781
          2.75229811668396
          0.5199907422065735
          2.200000047683716