Abstract
Human behaviour understanding (HBU) focuses on comprehending and analyzing human behaviours and it is comprised of diversified component tasks, e.g., action recognition, action prediction, pose estimation, etc. Due to its “all-encompassing” characteristic, HBU plays a pivotal role in many real-world application scenarios, such as urban video surveillance, disaster response, and self-driving technology. Therefore, developing an integrative deep learning intelligence to handle several tasks simultaneously is becoming urgent. However, the existing benchmarks typically focus on a single aspect of human behaviour understanding and provide specific training data for each task. Lack of a comprehensive dataset barricades optimization of a unified human behaviour understanding model. Additionally, the existing approaches tends to require task-specific architecture for heterogeneous output formats. Specifically, the action recognition task requires class predictions as outputs, while 3D pose estimation and motion prediction require human pose sequences. This discrepancy prevents the application of a uniform network architecture across different HBU tasks, highlighting two main challenges: (1) the absence of a comprehensive HBU benchmark and (2) the need to accommodate varied output formats across HBU tasks, thus obstructing the creation of a cohesive deep learning model for HBU. Therefore, with the aim of covering the unexplored areas, we first propose a large-scale drone-based HBU benchmark, which can provide multiple HBU tasks including action recognition, pose estimation, person re-identification and attribute recognition, with rich and elaborate annotations. For comprehensively analyzing human behaviors and actions from UAV videos, our proposed UAV-Human dataset provides annotations for four primary tasks - action recognition, pose estimation, person re-identification, and human attribute recognition - offering extensive and precise annotations. Specifically, to well provide these tasks, we engage a total of 119 distinct individuals with different ages, genders, and occupations to naturally perform various actions and poses in varied scenes during our dataset collection, which thus enables us to collect very diversified human actions and poses. Furthermore, to unite heterogeneous output formats for versatile HBU tasks, we propose a unified pose sequence modeling network, namely UPS, by considering text-based action labels and coordinate-based human poses as language sequences. Then by refining an auto-regressive transformer, we are able to obtain a cohesive output sequence for all HBU tasks. At the same time, to mitigate the effects of heterogeneity across different HBU tasks, a dynamic routing mechanism is also proposed to empower our UPS with the abilities in choosing which subsets of parameters to share for different tasks.