In the rapidly evolving world of singing voice generation, a significant hurdle remains: how to accurately assess the quality of these AI-generated vocals. While human listening tests are the gold standard, they’re often costly and time-consuming. Existing objective metrics, on the other hand, only capture a limited range of perceptual aspects. Enter SingMOS-Pro, a groundbreaking dataset designed to revolutionize automatic singing quality assessment.
Developed by a team of researchers including Yuxun Tang, Lan Liu, and their colleagues, SingMOS-Pro builds upon its predecessor, SingMOS. While SingMOS provided overall ratings, SingMOS-Pro expands its annotations to include lyrics, melody, and overall quality. This broader coverage and greater diversity make it a more comprehensive tool for evaluating singing quality. The dataset comprises 7,981 singing clips generated by 41 models across 12 datasets, spanning from early systems to recent advances. Each clip has been rated by at least five professional annotators, ensuring reliability and consistency.
The practical applications of SingMOS-Pro are vast. For music producers and audio engineers, it offers a more efficient and accurate way to assess the quality of AI-generated vocals. This could streamline the creative process, allowing for quicker iterations and refinements. For researchers, SingMOS-Pro provides a robust benchmark for developing and testing new evaluation methods. The dataset is now accessible on Hugging Face, inviting the broader community to explore its potential.
The researchers also delved into how to effectively utilize Mean Opinion Score (MOS) data annotated under different standards. They benchmarked several widely used evaluation methods from related tasks on SingMOS-Pro, establishing strong baselines and practical references for future research. This work not only advances the field of singing voice generation but also sets a new standard for assessing the quality of AI-generated vocals. As the technology continues to evolve, tools like SingMOS-Pro will be invaluable in pushing the boundaries of what’s possible in music and audio production.
