Seeing Through Uncertainty: A Free-Energy Approach for
Real-Time Perceptual Adaptation in Robust Visual Navigation

A neuroscience-inspired framework for real-time perceptual adaptation in robotic navigation.

Maytus Piriyajitakonkij¹ ³,Rishabh Dev Yadav¹, Mingfei Sun¹, Mengmi Zhang² ³, Wei Pan¹

¹ Department of Computer Science, The University of Manchester
² College of Computing and Data Science, Nanyang Technological University
³ Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)

Paper

Code

Problem Overview

In neuroscience, the Free Energy Principle (FEP) proposes that perception is an active, generative process: the brain continually predicts incoming sensory input and updates internal beliefs to stay accurate under uncertainty. This helps biological agents remain goal-directed even when observations are noisy, incomplete, or abruptly perturbed.

Motivated by this idea, we study robust visual navigation under sensory distribution shift (e.g., lighting changes, blur, rain, noise). We propose FEP-Nav, a real-time framework that adapts perception during inference without gradients so a navigation agent can remain functional under unexpected visual corruptions.

Figure 1: An example of the state-of-the-art model (DD-PPO) fails to navigate to goal in dimmed lighting condition.

Proposed Method

FEP-Nav improves robustness by combining top-down reconstruction and adaptive normalisation, which together approximate minimisation of the Variational Free Energy (VFE) objective. Under the Free Energy Principle, perception minimizes VFE, which can be decomposed into prediction error and Bayesian surprise.

The top-down decoder reconstructs a clean estimate of the visual observation from high-level features. This provides the agent with an internal prediction of what the environment should look like, helping reduce prediction error between expected and observed sensory inputs.

Meanwhile, adaptive normalisation updates the feature statistics of the visual encoder during inference to account for distribution shifts caused by corrupted observations. By aligning the corrupted feature distribution with the model’s prior representation, this process reduces Bayesian surprise, enabling real-time perceptual adaptation without gradient updates.

Together, these two mechanisms allow FEP-Nav to approximate VFE minimisation during inference, maintaining stable navigation performance under unexpected sensory conditions, as shown below.

Figure 2: FEP-Nav's single-step processing:
feedforward --> feedback --> feedforward

Demo

BibTeX

@article{piriyajitakonkij2024tta,

title={TTA-Nav: Test-time Adaptive Reconstruction for Point-Goal Navigation under Visual Corruptions},

author={Piriyajitakonkij, Maytus and Sun, Mingfei and Zhang, Mengmi and Pan, Wei},

journal={arXiv preprint arXiv:2403.01977},

year={2024}

}

Seeing Through Uncertainty: A Free-Energy Approach for Real-Time Perceptual Adaptation in Robust Visual Navigation

A neuroscience-inspired framework for real-time perceptual adaptation in robotic navigation.

Problem Overview

Proposed Method

Seeing Through Uncertainty: A Free-Energy Approach for
Real-Time Perceptual Adaptation in Robust Visual Navigation