Abstract
Music is a part of people’s daily lives. Listening to music on the radio, attend-ing concerts, dancing to music, and making music are just some of the musi-cal activities many people enjoy. While listening to music is basically the inter-pretation of sensations, making music requires the performer to create content. Hence, music-making requires a closed feedback loop to continuously evaluate and take control of actions to generate the intended sound. However, hearing loss limits the information available for interpreting and evaluating one’s own performance. The result is a gap in the feedback loop. The aim of this thesis is to investigate ways to support deaf people in music-making activities using sensory-substitution. My original contributions to scientific knowledge, of which most have been pub-lished in peer-reviewed venues, are 1. a comprehensive overview of mappings, practices, challenges, and trends in the area of music sensory-substitution systems based on a systematic literature review of 77 unique systems, 2. based on prior work and preliminary investigations, the definition of de-sign requirements and recommendations for music sensory-substitution systems that support deaf people in music-making activities, 3. the iterative development of a high-fidelity prototype (MuSS-Bits++) con-sisting of wearable, lightweight sensor–display pairs, and 4. the evaluation of the prototype in a controlled study and in music lessons, verifying that it fulfills all stated design requirements. The seven design requirements for MSSSs that support deaf people in music-making encompass (1) real-time feedback, (2) rhythm support, (3) capture of real-world audio sources, (4) adaptability, (5) ease of use, (6) allowing free limb movement, and (7) an enjoyable user experience. These requirements were nec-essary, as they guided the design, development, and evaluation of MuSS-Bits++. MuSS-Bits++ are the first technical realization that achieved all of the above requirements. The evaluation of MuSS-Bits++ revealed that deaf children per-ceive stronger cues through vibration. However, they may require more exposure to utilize this additional information to improve their performance in rhythm discrimination and reproduction tasks. In the future, music sensory-substitution systems will be able to adapt in many aspects to the user’s need (e.g., feedback modality, feedback location, and feedback intensity) and provide interfaces for users to optimize their experience by customizing mapping strategies.