Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction
Cunjun Yu*, Xiao Ma*, Jiawei Ren, Haiyu Zhao, Shuai Yi
SenseTime Research, National University of Singapore
Abstract
Understanding crowd motion dynamics is critical to real-world applications, e.g., surveillance systems and autonomous driving. This is challenging because it requires effectively modeling the socially aware crowd spatial interaction and complex temporal dependencies. We believe attention is the most important factor for trajectory prediction. In this paper, we present STAR, a Spatio-Temporal grAph tRansformer framework, which tackles trajectory prediction by only attention mechanisms. STAR models intra-graph crowd interaction by TGConv, a novel Transformer-based graph convolution mechanism. The inter-graph temporal dependencies are modeled by separate temporal Transformers. STAR captures complex spatio-temporal interactions by interleaving between spatial and temporal Transformers. To calibrate the temporal prediction for the long-lasting effect of disappeared pedestrians, we introduce a read-writable external memory module, consistently being updated by the temporal Transformer. We show that with only attention mechanism, STAR achieves the state-of-the-art performance on 5 commonly used real-world pedestrian prediction datasets.
Paper
Cunjun Yu*, Xiao Ma*, Jiawei Ren, Haiyu Zhao, Shuai Yi (*equal contribution)
Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction
BibTex
@inproceedings{
YuMa2020Spatio,
title={Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction},
author={Cunjun Yu and Xiao Ma and Jiawei Ren and Haiyu Zhao and Shuai Yi},
booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
month = {August},
year={2020}
}
Spatial and Temporal Transformers
STAR has two main components, Temporal Transformer and Spatial Transformer. (a) Temporal Transformer treats each pedestrians independently and extracts the temporal dependencies by Transformer model (h is the embedding of pedestrian positions, Q, K and V are the query, key, value matrix in Transformers). (b) Spatial Transformer models the crowd as a graph, and applies TGConv, a Transformer-based message passing graph convolution, to model the social interactions (mi→j is the message from node i to j represented by Transformer attention)
STAR
In STAR, trajectory prediction is achieved completely by attention mechanisms. STAR interleaves spatial Transformer and temporal Transformer in two encoder blocks to extract spatio-temporal pedestrian dependencies. An external read-writable graph memory module helps to smooth the graph embeddings and improve the consistency of temporal predictions. The prediction at T_obs + 1 is added back to history to predict the pedestrian poses at T_obs + 2.
Experiments
