4 Replies Latest reply on Mar 1, 2010 4:22 PM by bengfarrell

    Coordinate System on Sound Data



      I just started getting into pixel bender over the last couple of nights.  Basically I want to do the equivalent of downsampling and smoothing an image - but with audio data.  Basically I want to get the sound sample data down to a manageable data set to draw a waveform.


      So, with an image, I can get the pixel values of the 8 pixels around the center 9th pixel.


      I haven't attempted this on sound sample data yet,  as I'm trying to visualize what I need to do.  How is the coordinate system handled with a byte array instead of an image?  Is it something like a one pixel high image, where the width is the width of the byte array?  I'm basically wondering if I need to worry about the y-axis at all.



        • 1. Re: Coordinate System on Sound Data
          Kevin Goldsmith Level 3

          If you are using ShaderJob you can pass data in as a 1 dimensional buffer, but that will gaurantee you some bad performance, because the player divies up the data to work on using rows.


          Ideally you want to pass in a 2 dimensional buffer and use the width and height of the buffer as parameters to let you index to the nearest samples left and right.


          In the mixer sample I posted on my blog, I used float4s as values in a 2D buffer feeding in the samples as Sample1Left, Sample1Right, Sample2Left, Sample2Right for even better performance.


          I have some more sophisticated audio processing code I wrote in Pixel Bender that I'm planning to release when I get a chance that uses multiple samples. If I have some time, I'll see if I can post that here.

          • 2. Re: Coordinate System on Sound Data
            bengfarrell Level 1

            Thanks so much, you rock!

            So I did see that blog post, and I was fully planning to do what you suggest and use a float4.  Unfortunately for me, your post (or anybody else's byte array post) doesn't do anything in the way of sampling surrounding data unless it's an image, so I had no idea how the coordinate system worked.   I may use a float2 first just to see if it works and graduate to a float4.


            thanks again!

            • 3. Re: Coordinate System on Sound Data
              bengfarrell Level 1

              OK - um....now that I'm digging into this on the Flash end, I seem to be getting a little more confused.



              I've been doing some more experiments from the Flash side with a simple NumberCruncher filter by Ryan Taylor (http://www.boostworthy.com/blog/?p=243).  I've been dealing with a lot of too large or too small errors for the input data.


              I read somewhere that the input image dimensions are maxed out at 8195x8195 pixels, but I've been finding that the width is maxed out at 8192 pixels in reality.


              So I expanded on Ryan Taylor's example because I want to maximize the data that goes in, eventually working with sound.extract():


              _input = new ByteArray();

              _input.endian = Endian.LITTLE_ENDIAN;


              for (var c:int = 0; c < 8192; c++ ) {



              _input.position = 0;


              var width:int = _input.length >> 2;

              var height:int = 1;


              _shader = new Shader(new NumberCruncher());

              _shader.data.src.width = width;

              _shader.data.src.height = height;

              _shader.data.src.input = _input;


              _output = new ByteArray();

              _output.endian = Endian.LITTLE_ENDIAN;


              _shaderJob = new ShaderJob(_shader, _output, width, height);

              _shaderJob.addEventListener(Event.COMPLETE, onShaderJobComplete, false, 0, true);




              This works, but 8192 seems to be the upper limit of floats I can pass into this kernel (where the input is image1).  I thought that I might be able to go in the vertical direction, and make my height bigger to expand the number of floats I can pass in.


              So I thought that if I wanted to have 8192 * 2 floats, I could simply make the height of the input image as 2.  And if I wanted 8192 * 3 floats, I could make the input image's height 3, and I could do this all the way up to a height of 8192 to have 8192 * 8192 floats.


              Unfortunately, I'm must be misunderstanding something because even 8192 * 2 floats and a height of 2 gives me "Error #2177: The Shader input src is too large."


              If that had worked, I had further planned on changing the number cruncher to integrate some of your advice.  Use an image4 as the input to pass in 4x the input data for each pixel (four floats instead of 1), using bracket access in PixelBender to adjust each value in the array.


              Does this make sense?  Can you help me understand why I can't make the height bigger to accept more data?




              • 4. Re: Coordinate System on Sound Data
                bengfarrell Level 1

                OK I'm sorry....I'm an idiot!  I made the tied the width to the byte array length, and it made the width 2 * 8192 and that was obviously too much.

                So it seems to process the data OK with an increased height and byte array length.


                Hopefully nobody spent any time looking at this...