Real-Time Lip sync in Flash · Nov 13, 01:26 AM

On the left is lip-syncing based on pre-processed (voice recognition) phoneme mapping. On the right is real-time lip syncing utilizing the AS3 function computeSpectrum. Both use almost the same visual phoneme models, except the real-time version on the right currently uses 1 less phoneme.

The pre-processed version on the left uses a text-based voice recognition algorithm, which can theoretically be extremely accurate, but it does not produce perfectly aligned results. The results could also be maligned because they are not being rendered with the correct timing. Either way, it is a comparison of two far from perfect implementations of lipsyncing.

Which one do you like better?

The phoneme set I used as a guide.

I got the idea for real-time syncing from Second Life.

— Pickle

---

Comment

  1. The one on the right is better, it is synced better and the one on the left never closes his mouth :P

    — Siemer · Nov 13, 03:17 AM · #

  2. The One on the Right is better. It still needs some work. Smaller size might help.

    — David J McClelland · Dec 1, 07:13 AM · #

  3. i had a similar idea with using spectrum analyzing to move the mouth, cool to see it working!

    — john webber · Dec 12, 07:41 AM · #

  4. Sweet! how do you find the phonemes in the spectrum? Can we get the source-code?

    — Moose · Feb 5, 10:05 AM · #

  5. Moose
    The algorithm is sort of an approximation. There is a good but vague explanation if you click the Second Life link above. I’ll look into releasing the source…

    — Pickle · Feb 5, 09:06 PM · #

  6. So cool! Would be very interrested in source code for this.

    — Morten · Dec 10, 02:09 AM · #

Textile Help