Camera LiDAR Calibration

In mobile robotic systems, even small timing mismatches between sensors can lead to severe geometric inconsistencies. When a camera frame and a LiDAR scan are captured at slightly different moments, any motion during that interval causes the two sensors to disagree on where objects are located in 3D space. This appears as a “ghosting” effect when LiDAR points are projected onto camera images, significantly degrading perception and mapping performance. The goal of this project was to estimate and correct this timing offset to achieve accurate sensor alignment.

To measure how well the sensors are synchronized, I designed a multi-stage alignment pipeline that converts raw camera data into an optimization-friendly representation. Camera images were first processed to extract high-contrast geometric boundaries, producing binary edge maps that highlight structural features in the scene. These edge maps were then converted into smooth cost maps using a Distance Transform, where pixels farther from an edge incur higher penalties while pixels lying directly on an edge have zero cost. This formulation allows the system to penalize LiDAR points based on how far they deviate from expected camera geometry.

Temporal calibration was performed by evaluating alignment quality across a range of candidate timing offsets. For each offset, LiDAR points were projected into the camera frame and scored using the distance-transformed edge map. By iterating through possible offsets, the system identified the timing that maximized geometric consistency between the two sensor modalities. Visual inspection clearly showed this effect: when the offset was incorrect, projected LiDAR points drifted away from camera edges, while at the optimized offset, the projected geometry closely aligned with image contours.

The final system reduced temporal jitter to approximately 70 milliseconds, significantly improving cross-sensor alignment. By fusing IMU data with raw LiDAR measurements, I densified the point clouds to approximately 34,000 points per scan, providing the geometric resolution necessary for reliable calibration. The combination of grid search and efficient Distance Transform lookups enables rapid calibration without relying on specialized hardware synchronization, making the approach practical for real-world robotic deployments.