Combining Visible and Infrared Spectrum Imagery using Machine Learning for Small Unmanned Aerial System Detection

This research work proposes combining the advantages of the long-wave infrared (LWIR) and visible spectrum sensors using machine learning for vision-based detection of small unmanned air systems (sUAS). Utilizing the heightened background contrast from the LWIR sensor combined and synchronized with the relatively increased resolution of the visible spectrum sensor, a deep learning model was trained to detect the sUAS through previously difficult environments. More specifically, the approach demonstrated effective detection of multiple sUAS flying above and below the treeline, in the presence of birds and glare from the sun. With a network of these small and affordable sensors, one can accurately estimate the 3D position of the sUAS, which could then be used for elimination or further localization from more narrow sensors, like a fire-control radar (FCR).


Summary Video

Complete Detection Videos

Comparison between LWIR+RGB, LWIR, and RGB predictions - single drone case - confidence 50%

Comparison between LWIR+RGB, LWIR, and RGB predictions - multiple drones case - confidence 50%


Paper published at The 2020 SPIE Defense + Commercial Sensing: Automatic Target Recognition XXX:

arXiv preprint:

Winner of the "Best Student Paper" award. More info:

Download our Dataset

Link to dataset:

Thermal camera calibration files:

Example MATLAB files on how to use the calibration files to combine RGB and LWIR frames (by Junyan Cao, NUAA):

The dataset contains images from three synchronized RGB, long-wave infrared (LWIR), and near-infrared (NIR) cameras. There is a folder for each data collection condition/day, and for each condition/day, there will be more three folders: NIR, RGB, and THRM. Inside each folder, there will be two separate ones with images and labels. Most of them have labels for both THRM and RGB at different frames.

For any questions, please contact Vinicius Goecks. Most up to date contact information can be found at