Abstract
Machine learning has become a central decision-making tool in various applications, ranging from computer science, operations research, finance, biology etc. This is achieved in part, due to growing compute power that feeds increasingly complex models such as deep neural networks, sophisticated algorithms competing against one another and abundant data availability. This raises interesting questions about the efficacy and the efficiency of learning algorithms, which we aim to investigate in this thesis. Notably, we tackle some fundamental questions in deep learning, multiagent game theory and optimization. In deep learning, we investigate the role of “depth" by understanding the following question: Are there functions that can be expressed by deep networks that cannot be efficiently represented by shallow networks?. This is required to obtain a handle on the trade-offs between deep and shallow networks essential to understanding the cost of training such networks. Game theory may be seen as a framework to understand competing algorithms that produce the state-of-the-art performance rivaling human abilities in image recognition, multiplayer games etc. However, it is known even in simple games (such as Rock-Paper-Scissors), learning to play the game by repeated interaction with the competitor may lead to “complex" (formally chaotic) behavior. To this end, we address the following question: Are there conditions in which multiagent systems exhibit simple behavior? Finally, optimization is the cornerstone of machine learning. The performance of optimization algorithms is the key to understanding efficacy and efficiency, particularly when there is data truncation, since this is a common occurrence as data is collected by various sources, unknown to the modeler. In this thesis, we investigate the performance of the Expectation-Maximization algorithm for a two-component Gaussian mixture model (which is used in clustering applications). In this context we ask the following question: Can we provide guarantees for ML algorithms under data truncation? The answer to all the aforementioned questions has a surprising and deep connection to the theory of dynamical systems. This thesis explores these connections in detail and shows how various tools and techniques from dynamical systems are essential to address the challenges in modern machine learning.