14 Replies Latest reply: Jul 23, 2012 6:03 AM by sebrame RSS

    Auto Speech Alignment for foreign language dubbing

    sebrame Community Member

      I just watched the excellent "How to use Auto Speech Alignment in Audition" video on Adobe TV, and noticed that he mentions that ASA can be utilized to add foreign language dubs to a video. I find myself in that very process at the moment, and would really like to hear how ASA can help, but all the video mentions is the fact that it supposedly can.

        • 1. Re: Auto Speech Alignment for foreign language dubbing
          ryclark Community Member

          Have you tried doing it to see (hear) what happens?

          • 2. Re: Auto Speech Alignment for foreign language dubbing
            sebrame Community Member

            Not yet. I guess I could give it a shot. It's just that it will be comparing 2 radically different spectral patterns between the English and whatever language I'm dubbing over, and I wouldn't see how that would be possible.

            • 3. Re: Auto Speech Alignment for foreign language dubbing
              SteveG(AudioMasters) Community Member

              I have to say that I think you're correct. As far as I can make out, to a first approximation the process looks for comparable amplitude envelopes, and attempts to align these. This is fine when a revoice by the same actor is happening, but when the language, and inevitably the words and word shapes are changed, I would have thought you stood absolutely no chance with it at all.

               

              *update* I've now looked at the video, and listened to the claims. Since the whole thing is about stretching and shrinking the audio, then it can only really be achieved by amplitude envelope start time alignment wrt the reference channel - I think that the 'spectral analysis' claim is very likely to be wide of the mark, along with the foreign language overdub claim. Indeed, if spectral analysis was to be applied to completely different words in an attempt to align them, it would be likely to screw things up even more, not improve them - stands to reason, really.

               

              I blame a little too much presenter enthusiasm for this.

              • 4. Re: Auto Speech Alignment for foreign language dubbing
                sebrame Community Member

                "I blame a little too much presenter enthusiasm for this."

                 

                I'd agree, except that he is reading a script that I'm certain was written by Adobe, or at least 'approved' by Adobe. I can certainly understand the ability to stretch or shrink a foreign language clip to fit a previously cut English clips space, but that's nothing new, and certainly not a part of the Auto Speech Alignment tool.

                 

                The video sure did get my hopes up though.

                • 5. Re: Auto Speech Alignment for foreign language dubbing
                  SteveG(AudioMasters) Community Member

                  sebrame wrote:

                   

                  I'd agree, except that he is reading a script that I'm certain was written by Adobe, or at least 'approved' by Adobe.

                  I wouldn't bank on that! Since one of the Adobe 'evangelists' is a friend of mine, I asked him a while back how he does these videos (not this one - it's not him), and he assured me that he just 'wings' them. What generally happens is that they practice these presentations at exhibitions, etc and rely very much on the material they've presented to guide them as to what to say next - and they've all got the gift of the gab. Generally if something's not thought perhaps to be germane to the particular presentation, it hardly matters - except perhaps in this case, when it might.

                   

                  So basically, as far as I can make out, Adobe just relies on the integrity of their presenters - and hey, nobody's perfect...

                   

                  But in this case, you should try it, and see what happens - I think we'd all be very intrigued to know.

                   

                  If you do a search, you'll find a thread here where just before the release, I auto-aligned two sung lines that were almost identical, and even though that worked better than I thought it would, it wasn't perfect for this at all. One of the things that the devs said before release was that this feature wasn't intended to be used on pitched vocals, and as far as that's concerned, that pretty much slams the lid on there being a spectral analysis involved. I'd give you a link to the thread, but the search engine here won't find it...

                  • 6. Re: Auto Speech Alignment for foreign language dubbing
                    Charles VW Adobe Employee

                    I suggest also using the "Comments" section for the page that the video is on as well.  This may get more eyes here, but it _should_ get the eyes of the video's producer there.

                     

                    Anyways, the technology, does use spectral information in getting the alignment.  This is how it can achieve better results sometimes when compared to older existing technologies for automatic alignment.  For example, a waveform envelope may not be able to tell sibilance from a vowel sound when comparing an overdub vs the original field audio.  However, this is easy when looking at the spectral information and it can achieve a tighter alignment for that.

                     

                    That is, of course, assuming that the same words are being spoken in a similar way between the two clips.

                     

                    As for two different languges, I think it is a "Your Mileage May Vary" situation.  I think the intent of the suggestion is probably hoping that this feature can save you time if you wanted to get better lip-sync.  However, one can easily imagine that if the dubbed audio is stretched or squeezed too far that it (A) won't sound natrual, but more importantly (B) disrupts the performance of the voice actor.  On the other hand, if it is just doing micro adjustments it can make the timing better match that of the original mouth movements because it is making it match the timing of the original audio.

                     

                    I would suggest people with actual real-world content to try it out and preferebly post a video of your results (either positive or negative).  That would be a good way to further the discussion.

                    • 7. Re: Auto Speech Alignment for foreign language dubbing
                      sebrame Community Member

                      "and he assured me that he just 'wings' them."

                       

                      Well, just great. I'm also friends with one of the evangelists, and I'd always considered what they said as the 'last word'.

                      • 8. Re: Auto Speech Alignment for foreign language dubbing
                        sebrame Community Member

                        At any rate, comparing spectral patterns of diffeent languages cannot work. And lip-syncing different languages can't work either. I fully understand this, and don't expect any software to ever be aable to do this. However, when a software company makes the claim that it can, I tend to get my hopes up.

                         

                        However, ASA works very well for what it was intended, ADA and alignment of audio recorded seperately on location. Scarily well, I might add. We've just recently started working with a DSLR, requiring seperate audio, and I was fully prepared to purchase a 3rd party app to handle this until I attended NAB in the spring and saw that Audition could handle it alone.

                        • 9. Re: Auto Speech Alignment for foreign language dubbing
                          SteveG(AudioMasters) Community Member

                          Charles VW wrote:

                           

                          Anyways, the technology, does use spectral information in getting the alignment.  This is how it can achieve better results sometimes when compared to older existing technologies for automatic alignment.  For example, a waveform envelope may not be able to tell sibilance from a vowel sound when comparing an overdub vs the original field audio.  However, this is easy when looking at the spectral information and it can achieve a tighter alignment for that.

                          Hmmm.... I suppose that using a sibilance detector is using 'spectral information'... but that's not really what most people would understand by 'using spectral frequency data', which is the term he used. I think that really is quite misleading.

                           

                          To be fair, it does make sense to give yourself an edge (horrible pun intended), but I can still only see this screwing up pretty much any foreign language dub, same as sebrame does. My other experiments with it pretty much tie up with his - it works spectacularly well at what it does!

                           

                          And that demo video's been edited. Firstly, I should say that I don't understand why he split the clips - the auto-alignment should work fine with them just the way they are, shouldn't it? But having selected one, and just the first clip to align it to, the end result has both clips present and played in it...

                          • 10. Re: Auto Speech Alignment for foreign language dubbing
                            _durin_ Employee Hosts

                            We recently posted a interview with the creator of our Speech Alignment algorithm at http://blogs.adobe.com/insidesound/2012/07/interview-with-the-creator-of-the-automatic-spe ech-alignment-feature-in-audition-cs6.html where goes into a bit of detail about this.  I've spoken to him about other functionality we might be able to take advantage of within the algorithm, and a better foreign dubbing experience is at the top of my personal list.  Right now, the tool will match the duration of speech well, and depending on the contents, may align several dominant peaks.  We may be able to expose a toggle that essentially tells the algorithm the clips do NOT match content, and enable alignment differently in a way that might benefit foreign language dubbing.

                             

                            So the Adobe TV video is correct, but perhaps overselling this aspect of the tool right now (although just mentioning it to be a possible use case may be underselling it!)  I would love to get more feedback on everyone's results with Automatic Speech Alignment, and what sorts of functionality you'd like to see included.

                            • 11. Re: Auto Speech Alignment for foreign language dubbing
                              SteveG(AudioMasters) Community Member

                              _durin_ wrote:

                               

                              We recently posted a interview with the creator of our Speech Alignment algorithm at http://blogs.adobe.com/insidesound/2012/07/interview-with-the-creator- of-the-automatic-speech-alignment-feature-in-audition-cs6.html where goes into a bit of detail about this.

                               

                              And it says:

                               

                              Ellen: So this feature doesn’t do anything to help with overdubbing in a completely different language?

                              Brian: Not yet! But we’ve been looking at it.

                               

                              I think it's a bit of a tall order. Half the time, you're talking about dubbing to or from languages where the basic sentence structure is completely different - the verbs come in different places, and the numbers of words per sentence vary too. I would have thought that the last thing you'd be wanting to do is to temporally distort something spoken naturally in the local vernacular to fit that - it's almost bound to sound 'better' under these circumstances if it's left alone. Foreign language dubs make an excellent case for subtitling most of the time!

                               

                              Have to say though that I would have cheerfully paid some serious money for a tool that did what this one does back in the days when I was editing video.

                              • 12. Re: Auto Speech Alignment for foreign language dubbing
                                sebrame Community Member

                                "Foreign language dubs make an excellent case for subtitling most of the time!"

                                 

                                I need to forward this interview to our clients. They didn't listen to us when we said that!

                                • 13. Re: Auto Speech Alignment for foreign language dubbing
                                  SteveG(AudioMasters) Community Member

                                  sebrame wrote:

                                   

                                  "Foreign language dubs make an excellent case for subtitling most of the time!"

                                   

                                  I need to forward this interview to our clients. They didn't listen to us when we said that!

                                  Er, that's not in the interview - that was me... Brian just said that they were looking into it, and wisely stopped there (or Ellen edited it).

                                  • 14. Re: Auto Speech Alignment for foreign language dubbing
                                    sebrame Community Member

                                    Sorry. Still...the words contain vast amounts of wisdom.