Abstract
A new paradigm of software, machine learning (ML) has gained significant attention in recent years. Machine learning is immensely successful at tasks previously untouched by computer science. This new paradigm nonetheless faces more pragmatic challenges as it becomes a common feature of software infrastructure in production. To alleviate these challenges, we put forward the thesis: Machine learning software should be tested thoroughly before this software is deployed into production. This thesis aims to study and formalise some of the properties of machine learning software that need to be tested. Additionally, we propose techniques to automatically find violations of these properties and methods to mitigate the detected violations. By leveraging input formalisations, this thesis proposes a series of automated techniques to discover the violation of these properties, thus concretely highlighting flaws in implemented ML systems. In this dissertation, we explore the testing of the functional correctness and fairness of machine learning models. Leveraging the bugs found in the models, we also propose several techniques to localise and mitigate the faults. This body of work explores various sub-fields of machine learning, including models dealing with tabular data, natural language, speech recognition and computer vision. The techniques proposed in this thesis have found over 100,000 fairness and functionality bugs in leadingmachine learning models from Google, Amazon, IBM, Microsoft, and AllenNLP. This dissertation also serves as a reminder of the very real risks of transferring decision making to machine learning models. Specifically, we urge practitioners to consider the societal, economic and representational risks that come with deploying untested machine learning models in production systems. Such models have in the past caused various socio-economic harms, and have also led to the loss of life. In summary, we would like to encourage the exhaustive testing of machine learning models before they are deployed for meaningful decision making.