Lipsync node

Lipsync takes a video of a person and a separate audio clip, and redraws the mouth so it matches what is being said. The rest of the shot stays as it was.

It needs both. Wire the clip into the video input and the speech into the audio input. The audio usually comes from a Voice node, but any audio file works.

1. Wire your video into the node.
2. Wire your audio into the audio input.
3. Press Lipsync.

There are three models. Sync 3 is the newest and the default — reach for it on close-ups, profiles, and anything going to a client. Lipsync 2 Pro is stronger on beards and teeth. Lipsync 2 is the cheap one for rough passes. The two Lipsync 2 models add an expressiveness slider. Turn it up if the mouth moves too little, down if it looks overdone. They also have a setting for hands or objects crossing the face. Sync 3 does both of those itself, so neither control appears for it.

If the audio and video are different lengths chooses what happens when they don't match. More than one face on screen syncs only whoever is speaking when it is on. Something crosses the face watches for hands or objects, which is slower.

With several people in the shot, Pick the speaker chooses the face yourself. Press it, scrub to a frame where that face is clearly visible, click the middle of the face (not the hair, neck or body), and save. The node shows when the speaker was picked, with Change and Clear. Only that face is lip-synced and everyone else is left as they were. A pick replaces the automatic speaker setting, which is hidden while one is set.

1. Wire your video and audio into the node.
2. Press Pick the speaker.
3. Scrub to a frame where the face is clear, click the middle of the face, and press Save selection.
4. Press Lipsync.

A pick belongs to the clip it was made on. Wire in a different clip, or crop the one you have, and the node asks you to pick again before it runs. Some clips can't take a pick. The run then stops before anything is charged and says why. That happens with a clip stored rotated or flipped (common on Android phones), one trimmed inside the file without re-encoding, or one with a changing frame rate. Re-export it as a standard MP4, or clear the pick and use the automatic setting. If the lip sync finds no face at the point you clicked, the run stops without a charge and says so. Press Change, click the middle of the face, and run it again.

The person in your source clip should already be talking. These models redraw an existing mouth movement — they do not start one. Give a still, closed mouth and you will get a small movement at best. When you generate the source clip, ask for someone speaking to camera.

It also works best when the face is clear, reasonably large in frame, and facing roughly towards camera. A face in deep shadow or turned away is hard to match. If there are several people on screen, pick the speaker, or turn on the automatic speaker setting. The automatic one is a guess, and on a quiet or still clip it can choose the wrong face.