In the ever-evolving landscape of music technology, a groundbreaking development has emerged that promises to redefine the boundaries of singing voice synthesis (SVS) and singing voice conversion (SVC). A team of researchers, including Jionghao Han, Jiatong Shi, Zhuoyan Tao, Yuxun Tang, Yiwen Zhao, Gus Xia, and Shinji Watanabe, has introduced a novel machine learning task called Non-Human Singing Generation (NHSG). This innovative approach aims to generate musically coherent singing with non-human timbral characteristics, addressing a growing demand in creative applications such as video games, movies, and virtual characters.
The limitations of existing SVS and SVC systems have been their restriction to human timbres and their inability to synthesize voices outside the human range. This is where NHSG steps in, encompassing non-human singing voice synthesis (NHSVS) and non-human singing voice conversion (NHSVC). The challenge, however, is significant. The scarcity of non-human singing data, the lack of symbolic alignment, and the wide timbral gap between human and non-human voices present formidable obstacles.
To overcome these challenges, the researchers propose CartoonSing, a unified framework that integrates singing voice synthesis and conversion while bridging the gap between human and non-human singing generation. CartoonSing employs a two-stage pipeline. The first stage involves a score representation encoder trained with annotated human singing. The second stage features a timbre-aware vocoder that reconstructs waveforms for both human and non-human audio.
The experiments conducted demonstrate that CartoonSing successfully generates non-human singing voices. Moreover, it generalizes to novel timbres and extends conventional SVS and SVC toward creative, non-human singing generation. This breakthrough opens up new possibilities for music producers, developers, and enthusiasts, allowing them to explore a broader range of vocal timbres and creative expressions.
The implications of this research are far-reaching. By unifying human and non-human timbres, CartoonSing paves the way for more immersive and diverse audio experiences in various media. It challenges the established norms of singing voice synthesis and conversion, pushing the boundaries of what is possible in music and audio technology. As we continue to witness the rapid advancement of machine learning and artificial intelligence, the potential for innovative applications in the music industry is immense. This research not only ignites meaningful debate but also inspires future developments in the field.
