Abstract
Music captioning, the task of generating descriptive text for musical snippets, is a multifaceted challenge in artificial intelligence, requiring the integration of musical theoretical understanding and semantic understanding. This thesis proposes a novel approach to music captioning, utilizing multi-task learning to guide the caption generation model with music features. Initially, the input audio is encoded using the MERT encoder (Yizhi Li et al., 2023), capturing both low-level acoustic features and high-level musical information. This encoded representation is then fed into a Language Model (LM) with auxiliary heads dedicated to various music-understanding tasks, including instrument detection, genre classification, mood inference, key detection, and vocal gender identification. Each auxiliary head employs a projector to translate the extracted features into language token representations compatible with the LM. These token representations are concatenated with the tokenised textual query passed to the LM, enabling the model to generate captions that encapsulate the nuances of the input audio. Importantly, the model’s architecture allows for the seamless integration of general music representations as well as music-feature-specific information, resulting in captions that are not only descriptive but also musically informed. A significant contribution of this research is the development of a comprehensive augmentation of the MusicBench (Melechovsky et al., 2023) dataset synthesized by extracting diverse music features using existing music feature extractors through a modular feature extraction system. This dataset serves as the foundation for training the proposed model, ensuring its adaptability even with limited data availability. Through experimentation and evaluation, the efficacy of the proposed approach is demonstrated, showcasing its capability to generate informative and contextually rich captions when music features are incorporated into the learning process. Beyond its immediate applications in music captioning, the proposed framework offers insights into the symbiotic relationship between music-feature understanding and semantic music understanding. By bridging the gap between music and language, this research opens avenues for innovative applications in music recommendation systems, music generation, and immersive multimedia experiences. The findings presented in this thesis contribute to the ongoing work in the field of artificial intelligence, music cognition, and multi-modal applications for large language models.