In a significant leap forward for real-time facial animation, researchers have developed a method to create high-quality, low-resource models that can run directly on devices, opening up new possibilities for game development and digital character creation. The breakthrough comes from a team led by Zhen Han, who tackled the challenge of training robust machine learning models for speech-driven 3D facial animation without relying on massive datasets.
Traditionally, creating high-quality facial animation models requires a vast amount of paired audio and animation data. To circumvent this limitation, recent studies have introduced large pre-trained speech encoders that can handle variations in audio input, allowing the facial animation models to generalize across different speakers, audio qualities, and languages. However, these models are often too large for real-time, on-device use, restricting their application to offline scenarios on dedicated machines.
The researchers employed a technique called hybrid knowledge distillation with pseudo-labeling. This approach leverages a high-performing teacher model to train much smaller student models using a large audio dataset. Unlike the pre-trained speech encoders, the student models consist solely of convolutional and fully-connected layers, eliminating the need for attention context or recurrent updates. This simplification drastically reduces the model’s memory footprint and computational requirements.
In their experiments, the team demonstrated that they could shrink the memory footprint to as little as 3.4 MB and reduce the required future audio context to 81 milliseconds, all while maintaining high-quality animations. This advancement paves the way for on-device inference, a crucial step towards creating realistic, model-driven digital characters that can be used in games and other interactive applications.
For music and audio production, this technology could revolutionize the way digital characters and avatars are brought to life in real-time. Imagine a virtual singer that can lip-sync and express emotions in real-time, responding to the nuances of the audio input. This could enhance live performances, music videos, and even virtual reality experiences, making them more immersive and engaging. Additionally, the low-resource nature of these models means they can be integrated into a wide range of devices, from smartphones to gaming consoles, without compromising performance. Read the original research paper here.
