Aerial Vision-and-Dialog Navigation (AVDN)
Aerial Vision-and-Dialog Navigation (AVDN)
Yue Fan, Winson Chen, Tongzhou Jiang, Chun Zhou, Yi Zhang, Xin Eric Wang
University of California, Santa Cruz
Abstract:
The ability to converse with humans and follow natural language commands is crucial for intelligent unmanned aerial vehicles (a.k.a. drones). It can relieve people's burden of holding a controller all the time, allow multitasking, and make drone control more accessible for people with disabilities or with their hands occupied. To this end, we introduce Aerial Vision-and-Dialog Navigation (AVDN), to navigate a drone via natural language conversation. We build a drone simulator with a continuous photorealistic environment and collect a new AVDN dataset of over 3k recorded navigation trajectories with asynchronous human-human dialogs between commanders and followers. The commander provides initial navigation instruction and further guidance by request, while the follower navigates the drone in the simulator and asks questions when needed. During data collection, followers' attention on the drone's visual observation is also recorded. Based on the AVDN dataset, we study the tasks of aerial navigation from (full) dialog history and propose an effective Human Attention Aided Transformer model (HAA-Transformer), which learns to predict both navigation waypoints and human attention.
Paper and Code
Paper (arXiv)
Source Code (simulator and model training)
Dataset Download
Test unseen (without ground truth)
Demo Videos
AVDN within our Simulator
Human-Human ANDH
in real world
HAA-Transformer model
on real-world ANDH task
@article{fan2022aerial,
title={Aerial vision-and-dialog navigation},
author={Fan, Yue and Chen, Winson and Jiang, Tongzhou and Zhou, Chun and Zhang, Yi and Wang, Xin Eric},
Please cite our paper as below if you use our work.
@article{fan2022aerial,
title={Aerial vision-and-dialog navigation},
author={Fan, Yue and Chen, Winson and Jiang, Tongzhou and Zhou, Chun and Zhang, Yi and Wang, Xin Eric},
journal={arXiv preprint arXiv:2205.12219},
year={2022}
} journal={arXiv preprint arXiv:2205.12219},
year={2022}
}