Logo image
Back
Gesture Enhanced Comprehension of Ambiguous Human-to-Robot Instructions
Conference proceeding

Gesture Enhanced Comprehension of Ambiguous Human-to-Robot Instructions

Dulanga Weerakoon, Vigneshwaran Subbaraju, Nipuni Karumpulli, Tuan Tran, Qianli Xu, U-Xuan Tan, Joo Hwee Lim, Archan Misra and ACM
Proceedings of the 2020 International Conference on Multimodal Interaction, pp.251-259
ACM Conferences
ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
21/10/2020
DOI: https://doi.org/10.1145/3382507.3418863

Abstract

Computer systems organization -- Embedded and cyber-physical systems -- Robotics Human-centered computing -- Human computer interaction (HCI) -- Empirical studies in HCI Human-centered computing -- Human computer interaction (HCI) -- Interaction techniques -- Gestural input Human-centered computing -- Human computer interaction (HCI) -- Interaction techniques -- Pointing
This work demonstrates the feasibility and benefits of using pointing gestures, a naturally-generated additional input modality, to improve the multi-modal comprehension accuracy of human instructions to robotic agents for collaborative tasks.We present M2Gestic, a system that combines neural-based text parsing with a novel knowledge-graph traversal mechanism, over a multi-modal input of vision, natural language text and pointing. Via multiple studies related to a benchmark table top manipulation task, we show that (a) M2Gestic can achieve close-to-human performance in reasoning over unambiguous verbal instructions, and (b) incorporating pointing input (even with its inherent location uncertainty) in M2Gestic results in a significant (30%) accuracy improvement when verbal instructions are ambiguous.

Metrics

1 Record Views

Details

Logo image