Readings:

Face mesh is a very interesting and creative ML model that enables creators to set facial expression as a control. There are “buttons” like nose, eye, mouth, etc.

Inspired by the project “Move Mirror” designed by Google Creative Lab, I want to make a gif mirror corresponding to users’ facial expression.

Firstly, I need to load all the videos in p5.js:

function preload() {
  Dance = createVideo("gif/Dance.mp4");
  Haha = createVideo("gif/Haha.mp4");
  Serious = createVideo("gif/Serious.mp4");
  Smile = createVideo("gif/Smile.mp4");
  Why = createVideo("gif/Why.mp4");
}

function setup() {
  Dance.size(640, 480);
  Dance.volume(0);
  Dance.loop();
  Dance.hide();
  
  Haha.size(640, 480);
  Haha.volume(0);
  Haha.loop();
  Haha.hide();
  
  Serious.size(640, 480);
  Serious.volume(0);
  Serious.loop();
  Serious.hide();
  
  Smile.size(640, 480);
  Smile.volume(0);
  Smile.loop();
  Smile.hide();
  
  Why.size(640, 480);
  Why.volume(0);
  Why.loop();
  Why.hide();
}

Then, I need to read the data points from the face mesh model and calculate the distance between two datas to control the gif:

// This is an example of mouth control, it's of similar logic for other control.
if (myResults && myResults[0]){
    console.log(myResults[0]);
    const Mouth = myResults[0].parts.mouth;
    for (let i=0; i<Mouth.length; i++){
      textSize (16);
      const mouthDistance = Math.abs (Mouth[14]._y-Mouth[17]._y);
      // console.log(mouthDistance);
      if (mouthDistance > 20){
        let smile = Smile.get();
        image(smile, 0, 0);
        }
      else if (mouthDistance <= 5){
        let serious = Serious.get();
        image(serious, 0, 0);
               }
      }
}

To set the parameter, using the console.log() function to observe changes in data of your mouth:

console.log(mouthDistance);

gif if you like:

Dance.mp4

Haha.mp4