6 Replies Latest reply on Jul 8, 2014 3:41 PM by Howard Wang4

    Real Time Pitch Shift on iOS


      Hello Everyone,


      I am building a video/audio playback application for iOS and am having some trouble.  I have already worked through most of my problems and now am stuck on something that is just killing me.  I am using Andre Michelle's pitch shift for managing the audio samples used on my application.  Unfortunately when playing back on an iOS device I am getting extrememly bad sound playback.  On the web and on the PC playback is smooth and clean.  However when I play it back on an iOS device, I am getting a studdering effect (processor power?).  I have read elsewhere that bytearray may be the cause of the trouble as it does not perform well on iOS, but there has to be a way to fix this no?  Any help on the subject would be very helpful.  I have also read that using vector arrays would be an option, but, that would be a lot of reprogramming and to be honest, i dont understand the vector arrays well enough to implement them.


      You can see an example of the app in action (i will only leave it live for a the next month or till I get a result that works).

      kindesigns.com/venti   - Click on the blue box to get to the problemed audio player.  It works fine here, on the ipad not so much.


      Here is a link to the pitch shift code I am using...



      I have been at this project for months and now am stopped in place with no way to finish for my client.

      Thanks in advance for any help...  really...  thanks. 

        • 1. Re: Real Time Pitch Shift on iOS
          sinious Most Valuable Participant

          Have you tried Adobe's pitch shift example?



          Aside that, Vectors are simply an array that is strongly typed. You tell it what kind of data the array can hold and because it expects to only ever hold this kind of data, the dynamic casting overhead of a typical 'Array' is removed.


          Here's a little info on Vector performance versus ByteArray:



          In that you can see the performance of ByteArray.writeDouble (or writeFloat) versus a Vector.<Number>. It's quite a performance leap.


          A vector is as easy to use as learning the simple syntax:

          var myVector:Vector.<Type> = new Vector.<Type>();


          In your case you'd want to use floating point so you would use a Vector.<Number>. So:


          var myNumberVector.Vector.<Number> = new Vector.<Number>();


          Now you can use it similar to an Array.




          Vectors are useful for type strictness. You will get an error if you put any other 'type' in it because it's not an Array:


          myNumberVector.push('Hello World'); // error


          The error is useful so you can track down if some invalid data is assigned.


          A simple huge loop can show you if the data is lengthy a vector under this extremely simple circumstance can be good compared to a (cheating) ByteArray performing a similar operation (I'm sure someone else has a better example test):


          New AS3 doc:


          import flash.utils.getTimer;

          import flash.utils.ByteArray;


          // references

          var v:Vector.<Number> = new Vector.<Number>();

          var b:ByteArray = new ByteArray();

          var i:int; // counter


          // start time

          var startTime:uint = getTimer();


          // loop 10 million times pushing the number 123.456 into the vector

          for (i = 0; i < 10000000; i++)





          // perform an addition of index + index+1 on vector 10 million times

          // (stop at 9,999,999 because we're adding i+1)

          for (i = 0; i < 9999999; i++)


                    v[i] += v[i+1];



          trace("End of Vector time: " + (getTimer() - startTime) + "ms");


          startTime = getTimer();


          // same thing, but ByteArray

          for (i = 0; i < 10000000; i++)





          // value read at position

          var fVal:Number;

          for (i = 0; i < 9999999; i++)


                    b.position = i * 4;

                    fVal = b.readFloat();

                    // since it's just a test and I know the value is the same

                    // 4 bytes away I'll just re-use for a 'best case scenario'

                    // but if I must reposition and readFloat twice performance

                    // goes down

                    b.writeFloat(fVal + fVal);



          trace("End of ByteArray time: " + (getTimer() - startTime) + "ms");


          Traces on my machine:


          End of Vector time: 1230ms

          End of ByteArray time: 1806ms


          By cheat I mean you can see the comment. I'm not even taking the time to read the next indexed number from the next 4 byte (32bit) position then resetting position back to add together the 2 different positions, I'm re-using the same value. If I do that, the time gets more distant:



          var fVal:Number;

          for (i = 0; i < 9999999; i++)


                    b.position = i * 4 + 4;

                    fVal = b.readFloat();

                    b.position = i * 4;

                    b.writeFloat(b.readFloat() + fVal);




          End of Vector time: 1224ms

          End of ByteArray time: 2476ms

          1 person found this helpful
          • 2. Re: Real Time Pitch Shift on iOS
            Authentic7 Level 1

            Yea i think i found much of the same information but have no idea where I should implement it.  I have seen a few people compare the speeds between vectors and arrays.  I just couldnt understand how i would adjust my code without breaking andres package.  I actually gave it a go but ended up making nothing work so I backed off and reverted.  I also remeber that byteArray was needed to stream.


            I am a 3D artist doing app development, so I am not a "pro" by any means at writing AS3.  Im getting there though =) this project has been a great teacher for me. lol.


            So, as far as I can tell, this...


                                l0 = _target.readFloat();

                                r0 = _target.readFloat();


                                l1 = _target.readFloat();

                                r1 = _target.readFloat();


            ...is where my problem lies....  Any idea what I would need to adjust to make it "vector"?  Again this is all real time read/write as the pitch shift is adjustable on the fly.


            --Full function is below... full package is here : http://blog.andre-michelle.com/upload/mp3pitch/MP3Pitch.as


            private function sampleData( event: SampleDataEvent ): void


                        //-- REUSE INSTEAD OF RECREATION

                        _target.position = 0;


                        //-- SHORTCUT

                        var data: ByteArray = event.data;


                        var scaledBlockSize: Number = BLOCK_SIZE * _rate;

                        var positionInt: int = _position;

                        var alpha: Number = _position - positionInt;


                        var positionTargetNum: Number = alpha;

                        var positionTargetInt: int = -1;



                        var need: int = Math.ceil( scaledBlockSize ) + 2;


                        //-- EXTRACT SAMPLES

                        var read: int = _mp3.extract( _target, need, positionInt );


                        var n: int = read == need ? BLOCK_SIZE : read / _rate;


                        var l0: Number;

                        var r0: Number;

                        var l1: Number;

                        var r1: Number;


                        for( var i: int = 0 ; i < n ; ++i )


                            //-- AVOID READING EQUAL SAMPLES, IF RATE < 1.0

                            if( int( positionTargetNum ) != positionTargetInt )


                                positionTargetInt = positionTargetNum;


                                //-- SET TARGET READ POSITION

                                _target.position = positionTargetInt << 3;



                                //-- READ TWO STEREO SAMPLES FOR LINEAR INTERPOLATION

                                l0 = _target.readFloat();

                                r0 = _target.readFloat();


                                l1 = _target.readFloat();

                                r1 = _target.readFloat();



                            //-- WRITE INTERPOLATED AMPLITUDES INTO STREAM

                            data.writeFloat( l0 + alpha * ( l1 - l0 ) );

                            data.writeFloat( r0 + alpha * ( r1 - r0 ) );


                            //-- INCREASE TARGET POSITION

                            positionTargetNum += _rate;


                            //-- INCREASE FRACTION AND CLAMP BETWEEN 0 AND 1

                            alpha += _rate;

                            while( alpha >= 1.0 ) --alpha;



                        //-- FILL REST OF STREAM WITH ZEROs

                        if( i < BLOCK_SIZE )


                            while( i < BLOCK_SIZE )


                                data.writeFloat( 0.0 );

                                data.writeFloat( 0.0 );

                                //trace("writing 0's");





                        //-- INCREASE SOUND POSITION

                        _position += scaledBlockSize;



            as you can see its a rather complicated function for a noobie =)  I cant even tell where it is told to actually make the sound play.  I know its there, but for the life of me, i cant tell when the data stream is actually converted to an audio stream.

            • 3. Re: Real Time Pitch Shift on iOS
              sinious Most Valuable Participant

              If you read the SampleDataEvent documentation you'll see that when this handler is used Flash will turn a Sound object (the mp3) into a stream that is fed through the handler function. In every event your handler function receives you have a chance to read and alter the data before Flash plays it through the Sound class. The data is supplied as a ByteArray to begin with.


              Read the description here to get a better understanding:

              http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/events/SampleData Event.html


              The change occurs above when you see the ByteArray being read, those values are altered and then written back (right under the comment "//-- WRITE INTERPOLATED AMPLITUDES INTO STREAM".


              A computer is just a calculator. It doesn't understand what it's processing is actually audio. Via the Sound class, it does know how to interperate the ByteArray into actual sound however. So your job is to alter the values of the sound properly.


              Note that the block size can have performance changing characteristics. They recommend using a larger block size like 8192 so the code executes less times over all the data. You can try adjusting the BLOCK_SIZE value of that class between 2048, 4096 and 8192 to see if it helps remove some of your performance issues.


              Being the format given by the Sound class works with is ByteArray and the work being done to the stream is extremely simple I think converting the data to vector just to perform the same operation would actually hurt your performance. If you're just doing pitch I don't believe it will help you. If the Sound class could hand the data over in a Vector then it would be a different story.


              Some of the object instantiation can be removed if you make the ByteArray being generated in that function class-level and then just assign it. Object instantiation in something that's excessively recursive (functions, loops, etc) is a bit expensive for a low power device like a phone.


              Here's a bit more information on the sample file posted above and how Adobe made the pitch seem higher (by reducing samples):


              1 person found this helpful
              • 4. Re: Real Time Pitch Shift on iOS
                Authentic7 Level 1

                So, still at it.  some but little progress.  I am changing the block size and as you suggested, it does in fact change the playback quality.  Unfortunately, it does not seem to erradicate the problem entirely.  The "pops" happen farther away when dealing with a larger block size when I up the Block Size to 8192.  I have also tried to change the bit rate of the audio samples from 192 to 92 figuring less data would help.  Unfortunately it didnt really do much of anything.  I was really hopeful that it would have a substantial effect, but alas, it did not.  Bummer.


                The last point you made...

                "Some of the object instantiation can be removed if you make the ByteArray being generated in that function class-level and then just assign it. Object instantiation in something that's excessively recursive (functions, loops, etc) is a bit expensive for a low power device like a phone."


                Can you explain this a bit further?  I am not sure what you mean. 


                Thanks a billion.

                • 5. Re: Real Time Pitch Shift on iOS
                  sinious Most Valuable Participant

                  Seems to me while the sound got a little better the extra CPU it freed up might be utilized by something else, putting you right back in the same position. You might want to try an isolated example project with nothing but the sound in there to see if the CPU is really overworked and causing this. If you only process sound and it doesn't have artifacts in that case you know you just need to look for a way to free up the processor more during pitch shifts in your app.


                  What I mean by instantiation is every time you use the function it instantiates a new ByteArray object. Little things like that can be removed by moving that to a class-member that's re-used each time the function is fired since it's fired so rapidly.




                  var _soundByteArray:ByteArray = new ByteArray();


                  function sampleData(e:SampleDataEvent):void


                       // just assign an already existing ByteArray, don't create

                       _soundByteArray = event.data;


                       // continue..



                  If a function is fired rapidly then you'll want to do everything possible to cut down on the amount it does. One of those things is unnecessary object instantiation. Every little bit helps.

                  1 person found this helpful
                  • 6. Re: Real Time Pitch Shift on iOS
                    Howard Wang4



                    The Adobe example is not really pitch shift. It's time scale to make the song play faster or slower.


                    Audio time-scale/pitch modification - Wikipedia, the free encyclopedia


                    A clear definition is pitch shift is in the following wiki:


                    Pitch shift - Wikipedia, the free encyclopedia


                    The complete example of pitch shift is here




                    That's a very CPU intensive job.