Abstract
For Deep Neural Networks (DNNs) to effectively navigate the demands of contemporary video-based applications, there are several lingering challenges that must be addressed in terms of: efficiency, reliability, and performance. This thesis explores dynamic neural networks as a method for advancing the practicality of DNNs in contemporary video-based applications, overcoming the existing limitations in these three aspects.
Firstly, to improve real-time efficiency and latency of online video understanding in real-world applications, we consider the general case where the device’s state and the available resources fluctuate over time, which is not explored by existing works. In the first work, we propose a dynamic System-status-aware Adaptive Network (SAN) that considers the device’s real-time state to provide high-quality predictions with low delay, leading to improved efficiency and robustness to fluctuations of the system status. Moreover, training such an agent on various types of hardware configurations is not easy as the labeled training data might not be available, or can be computationally prohibitive. To address this challenging problem, we propose a Meta Self-supervised Adaptation (MSA) method that adapts the agent’s policy to new hardware configurations at test-time, allowing for easy deployment of the model onto other unseen hardware platforms.
Secondly, we investigate the reliability and robustness of dynamic networks, which can greatly reduce computation redundancy without compromising accuracy by adapting their structures based on the input. In the second work, we explore the robustness of dynamic networks against energy-oriented attacks targeted at reducing their efficiency. Specifically, we attack dynamic models with our novel algorithm GradMDM. GradMDM is a technique that adjusts the direction and the magnitude of the gradients to effectively find a small perturbation for each input, that will activate more computational units of dynamic models during inference.
Thirdly, we enhance the performance on fine-grained video-based tasks with dynamic expert architectures. For instance, early action prediction aims to successfully predict the class label of an action before it is completely performed, which is a challenging task because the beginning stages of different actions can be very similar, with only minor subtle differences for discrimination. In the third work, we propose a novel Expert Retrieval and Assembly (ERA) module that retrieves and assembles a set of experts most specialized at using discriminative subtle differences, to distinguish an input sample from other highly similar samples. To encourage our model to effectively use subtle differences for early action prediction, we push experts to discriminate exclusively between samples that are highly similar, forcing these experts to learn to use subtle differences that exist between those samples.
Additionally, we design an effective Expert Learning Rate Optimization method that balances the experts’ optimization and leads to better performance. Additionally, we further incorporate spatio-temporal specialization into our experts to tackle fine-grained action recognition, which aims to successfully discriminate between action categories with subtle differences. In the fourth work, we design a novel Dynamic Spatio-Temporal Specialization (DSTS) module, which consists of specialized neurons that are only activated for a subset of samples that are highly similar. During training, the loss forces the specialized neurons to learn discriminative fine-grained differences to distinguish between these similar samples, improving fine-grained recognition. Moreover, a spatio-temporal specialization method further optimizes the architectures of the specialized neurons to capture either more spatial or temporal fine-grained information, to better tackle the large range of spatio-temporal variations in the videos. Lastly, we design an Upstream-Downstream Learning algorithm to optimize our model’s dynamic decisions during training, improving the performance of our DSTS module.