TTA-Nav: Test-time Adaptive Reconstruction for Point-Goal Navigation under

TTA-Nav: Test-time Adaptive Reconstruction

for Point-Goal Navigation under Visual Corruptions

Maytus Piriyajitakonkij¹, Mingfei Sun¹, Mengmi Zhang² ³, Wei Pan¹

¹ Centre for AI Fundamentals, The University of Manchester
² School of Computer Science and Engineering, Nanyang Technological University
³ Centre For Frontier AI Research, Agency for Science, Technology and Research (A*STAR)

Paper

Code

Problem Overview

A robot, with an end-to-end deep navigation model that maps sensory inputs to motor command, suffers performance degradation when visual corruption occurs. Our method namely TTA-Nav is created to mitigate the performance degradation and allows the robot to adapt to visual corruption in real-time. An example of the performance degradation is shown below. The navigation model trained with 2.5 billion frames fails to navigate when the light is dimmed.

Figure 1: An example of the state-of-the-art model (DD-PPO) fails to navigate to goal in dimmed lighting condition.

TTA-Nav

We propose a novel Test-Time Adaptation (TTA) method to tackle the aforementioned problem. Our method relies on the Top-down Decoder (TD) and Adaptive Normalization (AN) in the pre-trained navigation model. The trained navigation model receives a corrupted image, denoises image features by adaptive normalization, and sends the high-level features to TD. Then, TD reconstructs the cleaner image and feeds it back to the pre-trained navigation model. The trained navigation model does the second forward pass to output motor command, i.e., action.

Figure 2: TTA-Nav Overview

Figure 3: TTA-Nav single-step processing:
feedforward --> feedback --> feedforward

Emergence of Image Restoration

TD is trained on clean images with a mean squared error (MSE) objective function to reconstruct the images from the high-level features extracted by the navigation model. Without experiencing visual corruption during training, TD can surprisingly reconstruct cleaner images. The reconstructed images are used as surrogate observations by the navigation model and enhance the model's navigation performance.

Demo

tta-nav-demo.mp4

BibTeX

@article{piriyajitakonkij2024tta,

title={TTA-Nav: Test-time Adaptive Reconstruction for Point-Goal Navigation under Visual Corruptions},

author={Piriyajitakonkij, Maytus and Sun, Mingfei and Zhang, Mengmi and Pan, Wei},

journal={arXiv preprint arXiv:2403.01977},

year={2024}

}