*** Presentation in IEEE ICRA 2026 ***

VIDEO: TCB-VIO for IEEE Robotics and Automation Letters (R-AL).

Agile and mobile robotic systems often operate under strict power and processing constraints. As a result, there is an increasing demand for low-latency and low-power camera technologies. State-of-the-art algorithms utilizing conventional cameras typically achieve frame rates of 40–80 frames per second (FPS). These cameras rely on an architectural design where data is captured, digitized, and transferred to separate digital processing hardware for further computation. This traditional architecture, characterized by the modular separation of sensors and processors, can introduce significant latency and increase power consumption.

On-sensor vision processing presents a novel paradigm that addresses these challenges by co-locating sensors and processors on the same chip. Focal-plane sensor processor arrays (FPSPs) exemplify this approach, with each camera pixel equipped with a small processor capable of processing and sharing data with adjacent pixels (See Fig. 1). This tight integration of sensing and processing reduces power consumption and minimizes delays caused by communication overhead, making FPSPs highly suitable for real-world robotic applications.

Due to the limited die space on FPSPs, per-pixel memory is highly constrained, and processing is typically performed in analog. Analog computation has inherent limitations, including reduced numerical precision, circuit inaccuracies, thermal effects, and noise leakage. These constraints challenge implementing computer vision algorithms and can lead to noisy results. Integrating additional sensing modalities, such as inertial measurement units (IMUs), using visual-inertial odometry (VIO) algorithms can reduce the impact of the noise. However, adapting VIO algorithms to FPSPs requires a redesign of the processing pipeline to preserve the high-speed and low-power advantages of FPSPs.

Early VIO algorithms were filtering-based, using the Extended Kalman Filter (EKF), Unscented Kalman Filter (UKF), and the popular MSCKF, where landmark positions are marginalized from the state vector to reduce the computational complexity. Later, smoothing approaches emerged, optimizing states over a window with tools like GTSAM, leading to variants such as VINS-Mono, SVO 2.0, Kimera VIO, and ORB-SLAM3. Despite improved accuracy, they can suffer from inconsistencies and linearization errors. VIO methods can also be categorized as loosely coupled or tightly coupled. Loosely coupled methods process the two modalities independently and fuse them later. In contrast, tightly coupled algorithms jointly optimize visual and inertial data to achieve higher accuracy and robustness.

For conventional cameras, however, the additional computation required by tightly coupled VIO often reduces update rates and increases the latency between image capture and state update. FPSPs can mitigate this trade-off: their in-pixel parallel processing enables fast on-sensor computations, while their ultra-high frame rate shortens the delay between image captures. For FPSPs, several visual odometry algorithms have been proposed. To date, the only VIO algorithm is BIT-VIO, which is a loosely coupled method; no tightly coupled VIO approaches exist. This gap is addressed in this paper.

This paper presents the first tightly-coupled 6 degrees-of-freedom (DoF) VIO algorithm, designed for FPSPs. The algorithm, coined TCB-VIO, is a Tightly-Coupled VIO utilizing BIT-enhanced features. TCB-VIO processes high-speed, on-sensor binary edge images and feature maps using a novel binary-enhanced Kanade-Lucas-Tomasi (KLT) tracker. It is an extended and modified adaptation of the tightly coupled framework, OpenVINS.

The contributions of this paper are:

(I) First high frame-rate tightly-coupled VIO for FPSPs achieving a high frame rate of 250 FPS.

(II) The framework processes binary edge data and feature coordinates directly on-sensor, reducing computational cost. These outputs are then processed by the novel binary-enhanced KLT tracker at high frame rates.

(III) Real-world evaluations comparing against ROVIO, VINS-Mono, and ORB-SLAM3.

Parameter optimization for TCB-VIO. Each plot shows the effect of varying one parameter on median ATE, RTE. Dashed vertical lines denote the optimal setting for each parameter.

Overview of representative testing trajectories used in our evaluation, aligned with the performance metrics reported in Table II. All trajectories were executed under fast and hostile motions, as reflected by the high angular velocities (10–31 rad/s) reported in Table II. Ground-truth is shown in gray, with error mapping done for TCB-VIO. 

Comparison of TCB-VIO against the baselines ROVIO and VINS-Mono on trajectory #10, as reported in table II. The grey lines are the ground-truth, while the coloured lines are the estimated trajectories. Note that the error scale is about twice as large for ROVIO and more nearly two order of magnitudes larger for VINS-Mono. As indicated in table II, ORB-SLAM3 failed to produce an estimate of the trajectory.

TABLE II. Results for TCB-VIO (as well as two ablations) against the baseline systems (ROVIO, VINS-Mono, ORB-SLAM3) in indoor environments, with ground-truth provided by a Vicon motion capture system. TCB-VIO shows superior ATE in most of the trajectories, and RTE in all of the trajectories, demonstrating its effectiveness in precise frame-to-frame tracking even in hostile motions.

TABLE V. Start-to-end errors for TCB-VIO compared to baselines in an outdoor environment.

This research is supported by Natural Sciences and Engineering Research Council of Canada (NSERC). 

We would like to thank Piotr Dudek, Stephen J. Carey, and Jianing Chen at the University of Manchester for kindly providing access to SCAMP-5.