• Adobe After Effects Expressions

  AE – Optimizing Lip-Sync or Keyframe-Handling

  Posted by Bogdan Brak on September 23, 2012 at 12:14 pm

  Hey guys,

  I would really appreciate your help.
  I’m trying to find an easy and timesaving way for lipsync.
  By now I’m using a really simple way of doing this. It works pretty fine but I’m sure there’s a way to make it better.

  I have a vector mouth. it’s an oval. Lets say: 100px witdh and 20px height.
  I took the voice-file and converted it and did the “Audio into Keyframes”-Action.
  I pickwhipped the scale of the the mouth to the slider of the channels.

  The code looks like this:
  temp = thisComp.layer("Audioamplitude 2").effect("Beide Kanäle")("Schieberegler");

  [100,100 + (linear(temp,10,25,0,5))*100]

  So. In this case the height of the mouth changes depending on the volume value. For simple cartoon-style-animation it works quite good.

  What’s my problem?
  The “Audio-into-keyframes”-action creates keyframes on every single frame of the comp. Is it somehow possible to tell AE to take only a look on the peaks and zeros?
  Maybe for better understanding…
  If we take a look at the keyframes that were creates, it looks like an edgy wave/diagram. it also looks like a path createt with Illustrator with a lots of anchor points. Is it possible to simplify this path? to kick out all unneccesary anchors and make a smooth (not edgy) wave containing the most important anchors(keyframes/peaks).

  I could imagine, that this way the lipsync might look much smoother.

  Thanks in advance

  PS: working with AE cs6

  Bogdan Brak replied 13 years, 10 months ago 2 Members · 5 Replies
• 5 Replies

• 

  Bogdan Brak

  September 23, 2012 at 12:45 pm

  one more concrete idea how it should work (without knowing how to implement this)

  let’s say we have three diffrent states:
  1. no one speaking (if music value <10)
  The mouths’ height = 100%

  2. someone’s talking (if music >= 10 && < 20)
  The mouths’ height = 130%

  3. someone’s talking loud/screaming(if music >= 20)
  The mouths’ height = 160%

  The mouths’ height is ALWAYS trying to get down to 100%.
  Maybe this way:
  for(i = 100; i > 100; i–)

  F.e. If the height is at 140% because it’s in the third state it should count down to 100%.

  That would also mean, that after effects should NOT check the state every single frame. It must check it maybe every 500milliseccond.

  I hope you can follow me…
  I try to sum it up:

  The audiospectrum must be devided and mapped into three states.
  – no sound
  – sound
  – loud sound

  after effects must check the state every 500ms and not every frame.
  the mouths height should always try to close and animate the mouth if its value is bigger than 100%

  and… the whole thing should get an Easing.

  Any ideas how to implements this in AE?
  cheers

  and again…thanks in advance
• 

  Bogdan Brak

  September 23, 2012 at 1:10 pm

  ok…now I think I know how to explain it in a few words.
  The whole thing shoul act like a smooth VU-meter. It should not check the peak 30 times a secon. Maybe only 4 times. And after every check the animation/mouth is trying to come down to 100%.
• 

  Xavier Gomez

  September 24, 2012 at 11:09 am

  What you are describing (reading audio amp values at periodical time intervals) doesnt look too good, not very convincing motion. It can be written like this:

  var sc = transform.scale // the keyframe value of the scale
  var T = 0.50 // the period at which the audio amp is read
  var t = time-(time%T);
  var audioAmp = thisComp.layer(“Audioamplitude 2”).effect(“Beide Kanäle”)(“Schieberegler”).valueAtTime(t);

  var extra;
  if (audioAmp>=20) extra = 60;
  else if (audioAmp>=10) extra = 30;
  else extra = 0;
  sc +[0,easeOut(time-t, 0,T, extra, 0)];

  ————

  I would like to suggest another expression but for some reason it won’t copy in the browser; part of the code is being chunked out, even if i copy it in the “code” box. Sorry.

  Xavier.
• 

  Xavier Gomez

  September 24, 2012 at 11:21 am

  Finally got it to paste it correctly:

  // the parameters to be tweaked;
  var thresh1=10; var extra1 = 30;
  var thresh2=20; var extra2 = 60;
  var dampling = 0.5;

  var sc = transform.scale // the keyframe value of the scale
  var audioAmp = thisComp.layer(“Audioamplitude 2”).effect(“Beide Kanäle”)(“Schieberegler”);
  var dt = thisComp.frameDuration;

  var t=time;
  var val= audioAmp.valueAtTime(t);
  var val_old = audioAmp.valueAtTime(t-dt);

  while(val_old>val && t>0){
  t -= dt;
  val=val_old;
  val_old=audioAmp.valueAtTime(t-dt);
  }
  var extra;
  if (val>thresh2) extra = extra2;
  else if (val>thresh1) extra = extra1;
  else extra = 0;

  sc + [0, extra * Math.exp(-dampling*(time-t))];
• 

  Bogdan Brak

  September 24, 2012 at 11:47 am

  Hey great. Thanks. Got to play around a little bit with it.

  By the way. Yesterday I found a way.

  simply that code:
  temp = thisComp.layer("Audioamplitude 2").effect("Beide Kanäle")("Schieberegler");
if(temp < 8){
temp = 0;
}
[100 ,temp*20 +140]

  than I made the curve of the amp-values smoother. (There’s a function for this in AE / Window -> Smoothen)

  finally I added a little bit of the CC Motion blur effect.

  Acutally… it looks pretty good.

  But still got to check your code.

Reply to this Discussion! Login or Sign Up