Institute of Information Science, Academia Sinica

In recent years, neural language models such as GPT (Generative Pre-trained Transformer) have been widely applied to assist in various tasks related to multimedia content generation and understanding. However, the limitations and challenges of applying these models to musical data warrant careful consideration. In this talk, we address an intriguing music understanding problem: How can a neural network model analyze and represent the structure of music? We define this problem from the perspective of computational musicology and discuss related tasks such as boundary detection, segment labeling, and motif discovery in musical signals. Recognizing the unique characteristics of musical data, we examine the capabilities and limitations of general-purpose neural language models in these tasks. In particular, we emphasize that understanding the hierarchical structure of music cannot be reduced to a closed-set classification problem, but must instead be formulated as a form of context-dependent clustering. From an implementation standpoint, this requires a combination of supervised and self-supervised training strategies. Finally, inspired by these research findings, we will discuss the significance of musical structure discovery mechanisms in application scenarios such as music information retrieval and music generation.