Abstract: Thermal (long-wave infrared) sensors show great promise to enable autonomy in visually degraded environments and at night without active illumination. However, they fundamentally behave differently from visual spectrum cameras and require special processing. In this talk I will outline the fundamentals of these sensors, and our latest algorithms and results to enable autonomous robots to navigate and map with multi-spectral and thermal sensors. 

Abstract: Machine perception uses advanced sensors to collect information about the surrounding scene for situational awareness. State-of-the-art machine perception using active sonar, radar and LiDAR to enhance camera vision faces difficulties when the number of intelligent agents scales up. Exploiting omnipresent heat signal could be a new frontier for scalable perception. However, objects and their environment constantly emit and scatter thermal radiation, leading to textureless images famously known as the ‘ghosting effect’. Thermal vision thus has no specificity limited by information loss, whereas thermal ranging—crucial for navigation—has been elusive even when combined with artificial intelligence (AI). Here we propose and experimentally demonstrate heat-assisted detection and ranging (HADAR) overcoming this open challenge of ghosting and benchmark it against AI-enhanced thermal sensing. HADAR not only sees texture and depth through the darkness as if it were day but also perceives decluttered physical attributes beyond RGB or thermal vision, paving the way to fully passive and physics-aware machine perception. We develop HADAR estimation theory and address its photonic shot-noise limits depicting information-theoretic bounds to HADAR-based AI performance. HADAR ranging at night beats thermal ranging and shows an accuracy comparable with RGB stereovision in daylight. Our automated HADAR thermography reaches the Cramér–Rao bound on temperature accuracy, beating existing thermography techniques. Our work leads to a disruptive technology that can accelerate the Fourth Industrial Revolution (Industry 4.0) with HADAR-based autonomous navigation and human–robot social interactions.

Abstract: In this talk we present progress towards resilient robotic autonomy and the key role of thermal vision for robust localization and navigation. The presented results start from efforts linked to the DARPA Subterranean Challenge and proceed with recent investigations for perceptually-degraded autonomy especially in obscurants-filled GNSS-denied settings. In particular, we will first discuss efforts for thermal-inertial odometry and their fusion in a multi-modal SLAM framework. The different performances between direct and feature-based techniques will be presented. Subsequently, we will present results on factor graph-based fusion with LiDAR and inertial cues, alongside the ability of such a framework to seamlessly incorporate additional modalities such as mmWave radar. Finally, new results towards thermal vision-based safe navigation will be presented.  

Abstract: TBA

Abstract: Thermal-based perception is becoming increasingly essential in environments where conventional sensors, such as RGB cameras, often fail—particularly in darkness, dense smoke, or heavy rain conditions. Accurate intrinsic and extrinsic calibration is critical for enabling robust multi-sensor fusion in such challenging scenarios. This talk will firstly present our work on intrinsic and extrinsic calibration across thermal cameras, 3D LiDAR, 4D radar, and RGB cameras. To address the traditionally tedious and time-consuming nature of sensor calibration, we introduce a unified framework designed to streamline the process and reduce manual effort across heterogeneous modalities. The second part will discuss downstream applications made possible by the calibration, including thermal-based multi-modal perception tasks such as object detection and SLAM. Finally, will share insights from our ongoing research projects that aim to tackle perception challenges under adverse weather conditions, particularly in tropical and firefighting environments.