Abstract
The objective of speech enhancement is to remove the background noise from the corrupted audio signals. Traditionally, it is a supervised task, using the noisy audio as input and the clean audio as output. However, supervised learning creates some problems. Supervised learning requires matching pairs of clean and noisy data which is hard to obtain in real life, therefore, most datasets are created by adding noise to clean audio. However, there are almost infinite types of noise in real-life situations, the dataset can only cover a limited number. Even with data augmentation, the problem can be relieved a bit, but not fully solved. When facing unseen noise in the training phase, the performance of the model will shrivel.
In order to tackle these problems, this thesis proposes a novel method of based on the Denoising Diffusion Probabilistic Model (DDPM) [1] to fulfill speech enhancement tasks through self-supervised learning. Compared to traditional speech enhancement models, instead of parallel data of matching pairs of clean and noisy audio, the proposed model only requires clean data for training and is able to maintain its performance with seen and unseen noise. The proposed model utilized an overlooked property that the output of DDPM can be guided by a latent input. Therefore, instead of using a Gaussian noise as the input and setting the timestep to the maximum timestep and randomly generating an output that we cannot control, we use the noisy audio as the guidance and set the timestep to a relatively small number compared to the maximum timestep. The model also has an extra benefit that it has faster inferencing speed as unlike traditional DDPM-based models, it only uses part of the total timestep instead of all of it. Compared to CDiffuSE [2], the number of floating-point operations (FLOPs) of our model is significantly lower with a huge margin. Our model requires only 0.757G FLOPs for inferencing, while CDiffuSE requires 252G FLOPs1.
The model yields promising results. Based on the subjective and objective evaluation results, by comparing to the award-winning model, DCCRN [3], the model we proposed is capable of performing similarly to DCCRN.