Abstract
Multi-modal learning is of paramount importance in the development of artificial intelligence due to its ability to leverage diverse sources of information from multiple modalities, such as text, images, audio and other modalities, mimicking the way humans perceive and understand the world. This capability is significant because real-world data is often multi-faceted, requiring comprehensive understanding beyond a single type of data. However, effectively harnessing the rich information contained within various modalities presents significant challenges, including model reasoning capability, inference efficiency, prediction robustness, and the ability to continually adapt to new data streams. Specifically, existing multi-modal models often struggle to effectively address complex reasoning problems. Moreover, there is a lack of comprehensive and challenging benchmarks specifically designed to thoroughly test the reasoning capabilities of multi-modal models, which impedes efforts to enhance their reasoning abilities. In terms of inference efficiency, it is crucial for deploying multi-modal models in real-world applications where computational resources may be limited and response time is critical, such as in autonomous driving systems. Another important factor affecting multi-modal models’ performance is the prediction robustness, which measures the model ability to maintain high accuracy and consistency in its predictions across diverse data sets. Last but not least, the capability to adapt to new data streams is critical for multi-modal models when deployed in real-world applications. This adaptability allows a model to update its knowledge base and refine its algorithms based on new data without needing a complete retraining from scratch. With the aim of handling the above challenges in multi-modal learning, we first build a novel benchmark, aimed at evaluating and enhancing the capability of multi-modal models to perform causal inference and complex reasoning through the task of video question answering (video QA). The dataset comprises 10,080 videos and 62,535 QA pairs, containing six types of diverse reasoning tasks that evaluate different aspects of model reasoning ability. Additionally, we design an efficient video reasoning network that utilizes dynamic inference to improve computation efficiency while maintaining reliable video reasoning accuracy. Besides, to improve the prediction robustness of multi-modal models, we introduce a novel framework to enhance the prediction robustness of models in referring expression segmentation tasks, particularly when encountering novel compositions of learned concepts. To explore the capability of multi-modal models to continuously learn from new data streams, we propose an innovative network, which integrates multiple learning blocks within its architecture. This network is designed to dynamically select and update task-specific blocks based on the current reasoning task. This approach helps the system efficiently learn new tasks by utilizing previously learned experts and incorporating new ones as needed.