Foley
The plugin makes sound for a video clip from what happens in it, in time with the picture. A prompt is optional. Without one the sound follows the picture, and with the prompt can be a sound whose source is off screen.
Requirements
Linux, Windows, macOS. GPU or CPU: about 4 GB VRAM for the Small model and 6 GB for Large. On a CPU or Apple silicon it works but much more slowly. Python 3.11. Models: about 7 GB (Small) or 11 GB (Large).
Disclaimer
The plugin code is developed by Wunjo. It uses MMAudio V2 by its authors, together with BigVGAN v2, Synchformer and Apple’s DFN5B CLIP encoder. The MMAudio models are licensed under CC BY-NC 4.0, and the CLIP encoder under the Apple Machine Learning Research Model License. Both allow non-commercial use only, so the plugin and the sound it makes may be used for personal and research work, not commercially. Results depend on the source material and are not guaranteed.
How to use
Right-click on a video clip on the timeline. In the menu that appears, select “Artificial Intelligence” > “Foley”.
The effect will appear in the “Effects / Composition Stack” panel on the left side of the screen. Optionally describe in “Sound” what should be heard, and in “Without” what should not. Click “Generate into the bin” within the “Foley” effect and wait for the generation process to complete. Upon completion, the sound will appear on a new audio track under the clip and in the “Foley” folder of the bin. The clip itself is not changed.
In the plugin settings you can choose the Small or Large model, the number of steps, how closely to follow the prompt.
How to install
Download the .wmplugin file. In Wunjo Make, open “Settings” and then “Plugins”, and click “Add plugin…”. Select the downloaded .wmplugin file. The plugin will be copied into the application.
The plugin’s Python environment and model weights are downloaded automatically the first time you run it, so keep your internet connection active until the process is complete. You can also download the required Python libraries and models manually from the plugin settings. An internet connection is required for the initial download, but it is not needed for subsequent use.