In the rapidly evolving world of generative speech models, researchers are grappling with a new challenge: how to quantify and detect the gibberish that these advanced systems can sometimes produce. A team of researchers from the University of Oldenburg, led by Danilo de Oliveira, Tal Peer, Jonas Rochdi, and Timo Gerkmann, has been exploring this very issue. Their work is particularly relevant as generative models become increasingly capable of producing high-quality speech, making it more difficult to distinguish between human speech and machine-generated gibberish.
The team’s research focuses on non-intrusive quality and intelligibility assessment, which is crucial for curating large-scale datasets of real-world speech. As generative models improve, they can synthesize speech that sounds convincing but contains subtle or not-so-subtle errors, such as phoneme confusions or outright gibberish. Intrusive metrics, which compare generated speech to a reference signal, can spot these discrepancies. However, non-intrusive methods, which don’t rely on a reference signal, struggle with this task. The researchers aim to bridge this gap by leveraging language models in a fully unsupervised setting.
To tackle this problem, the team has published a dataset of high-quality synthesized gibberish speech. This dataset is designed to help develop measures for assessing implausible sentences in spoken language. Alongside the dataset, they’ve also released code for calculating scores from a variety of speech language models. This resource will be invaluable for other researchers and developers looking to improve the quality and reliability of generative speech models.
The practical applications of this research are significant. For music and audio production, generative speech models are increasingly being used to create realistic vocal tracks, voice-overs, and even entire dialogues. However, the presence of gibberish or phoneme confusions can ruin the immersive experience. By developing better methods to detect and quantify these artifacts, the researchers are helping to ensure that generative speech models can be used effectively and reliably in these creative fields. Moreover, their work could lead to improvements in speech-to-text technologies, virtual assistants, and other applications where accurate speech recognition and synthesis are crucial.
