Lip Sync
Syncs the lips of a person on screen to a new audio track. Dubbing into another language, replacing a line after the shoot, voicing existing footage. The plugin redraws only the mouth area, so facial expressions, head turns, gestures and everything else in the frame stay as they were in the source. It works best when the face is clearly visible and does not turn too far from the camera. Processing runs on the GPU and takes noticeably longer than the clip itself.
Requirements
Linux, Windows. 12 GB VRAM, GPU. Python 3.11. Models: ~3.8 GB download.
Disclaimer
The plugin code is developed by Wunjo. The models are made by third-party authors and come under their own licenses. Results depend on the source material and are not guaranteed. You are responsible for the content you create and for the consent of anyone whose face or voice appears in it.
How to use
Right-click on the video clip that contains at least one human face. In the menu that appears, select “Artificial Intelligence” > “Enable Face Detection”. Rectangular overlays will appear around the detected faces in the “Project Monitor”. These overlays are only used for face detection and will not appear in the final render.
Left-click on the face rectangle and select “Lip Sync” from the menu that appears. Then click on the video clip in the timeline. The “Lip Region” and “Lip Sync” effects will appear in the “Effects / Composition Stack” panel on the right side of the screen. If necessary, you can adjust the position of the “Lip Region” effect on the timeline to refine the face detection results.
In the “Lip Sync” effect settings, find the “Audio source” section and select the audio file you want to use for lip-syncing. If the required audio file has not been loaded yet, click the edit icon to load a new file. In the window that opens, select and load the desired file. You can also use this window to preview audio files that have already been loaded.
Once you have selected the audio file, click “Generate into the bin” and wait for the generation process to finish. When the process is complete, the generated file will appear on the timeline.
How to install
Download the .wmplugin file. In Wunjo Make, open “Settings” and then “Plugins”, and click “Add plugin…”. Select the downloaded .wmplugin file. The plugin will be copied into the application.
The plugin’s Python environment and model weights are downloaded automatically the first time you run it, so keep your internet connection active until the process is complete. You can also download the required Python libraries and models manually from the plugin settings. An internet connection is required for the initial download, but it is not needed for subsequent use.