In a groundbreaking development for music technology, researchers have unveiled a unified framework that bridges two fundamental yet inverse tasks in music information retrieval: expressive performance rendering (EPR) and automatic piano transcription (APT). This innovative approach, developed by Wei Zeng, Junchuan Zhao, and Ye Wang, promises to revolutionize how we interact with and create music, offering practical applications for both music production and education.
The researchers’ framework addresses a longstanding challenge in the field by jointly modeling EPR and APT. Expressive performance rendering generates expressive performances from symbolic scores, while automatic piano transcription recovers scores from performances. Prior work has tackled these tasks independently, but this new approach disentangles note-level score content and global performance style representations from both paired and unpaired data. This means that the system can learn to understand and separate the musical content from the way it is played, allowing for more nuanced and flexible music generation and transcription.
Built on a transformer-based sequence-to-sequence architecture, the framework is trained using only sequence-aligned data, eliminating the need for fine-grained note-level alignment. This makes the system more efficient and easier to train. To further automate the rendering process and ensure stylistic compatibility with the score, the researchers introduced an independent diffusion-based performance style recommendation module. This module generates style embeddings directly from score content, supporting both style transfer and flexible rendering across a range of expressive styles. In other words, the system can take a piece of music and generate a performance that matches the style of another piece, or it can render the same piece in different styles.
The practical applications of this technology are vast. For music producers, it offers a powerful tool for generating expressive performances from MIDI files, enhancing the creative process. For educators, it provides a means to create stylistically appropriate renditions of musical pieces, aiding in the teaching of performance techniques. Additionally, the system’s ability to transcribe performances into scores can be invaluable for musicians looking to preserve their work or for archival purposes.
Experimental results from both objective and subjective evaluations demonstrate that the framework achieves competitive performance on EPR and APT tasks. Moreover, it enables effective content-style disentanglement, reliable style transfer, and stylistically appropriate rendering. This means that the system not only performs well in generating and transcribing music but also excels in maintaining the stylistic integrity of the performances.
The researchers have made demos available at https://jointpianist.github.io/epr-apt/, allowing users to experience the capabilities of this innovative framework firsthand. As the field of music information retrieval continues to evolve, this unified approach to EPR and APT represents a significant step forward, offering new possibilities for music creation, education, and preservation. Read the original research paper here.
