Abstract
In computer science, arti?cial intelligence research is de?ned as the study of intelli-gent agents: any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals (From: Wikipedia contributors, 2018). Hu-man action recognition is a challenging and crucial problem in arti?cial intelligence since it is a key part in perceiving the environment. Herath, Harandi, and Porikli, 2017 provide a consolidated de?nition of action that it is the most elementary human-surrounding interaction with a meaning. Generally, crowd-sourced data from various sensors and multi-view cameras can be given as input. Thus, to identify the actions performed by humans, we could formulate the action recognition problem as a binary (or multi-class) classi?cation problem that outputs the action categories. In this dissertation, we propose multi-modal approaches that achieve human ac-tivity recognition from multiple views. In addition, we address the security problem of deep neural network (DNN) as recent work points out that adversary can craft ad-versarial perturbations to attack deep learning systems such as video analysis system (Hosseini et al., 2017). In particular, we ?rst build a multi-modal egocentric activity dataset. On top of that, we design both conventional machine learning and multi-stream deep learning approaches to achieve multi-modal activity recognition. Second, we describe a truly multi-modal model for solving large-scale classi?cation problem with text, audio and videos. Our solution ranked top 3% out of more than 600 teams in the YouTube-8M Video Understanding Challenge. Third, we design an adaptive tem-poral pooling module in deep learning framework to better modeling actions in third-person view activity recognition. Lastly, to improve the robustness of our proposed DNN-based activity recognition systems, we present a Saak-transform-based method to remove adversarial perturbations. Without harming the decision performance on clean images, our method outperforms state-of-the-art adversarial defense methods.