Combining Visible and Infrared Spectrum Imagery using Machine Learning for Small Unmanned Aerial System Detection

Vinicius G. Goecks, Grayson Woods, John Valasek
Texas A&M University

This research work proposes combining the advantages of the long-wave infrared (LWIR) and visible spectrum sensors using machine learning for vision-based detection of small unmanned air systems (sUAS). Utilizing the heightened background contrast from the LWIR sensor combined and synchronized with the relatively increased resolution of the visible spectrum sensor, a deep learning model was trained to detect the sUAS through previously difficult environments. More specifically, the approach demonstrated effective detection of multiple sUAS flying above and below the treeline, in the presence of birds and glare from the sun. With a network of these small and affordable sensors, one can accurately estimate the 3D position of the sUAS, which could then be used for elimination or further localization from more narrow sensors, like a fire-control radar (FCR).

Sensors

Summary Video

Complete Detection Videos

Comparison between LWIR+RGB, LWIR, and RGB predictions - single drone case - confidence 50%

Comparison between LWIR+RGB, LWIR, and RGB predictions - multiple drones case - confidence 50%

Publications

Paper published at The 2020 SPIE Defense + Commercial Sensing: Automatic Target Recognition XXX: https://spie.org/Publications/Proceedings/Paper/10.1117/12.2557442

arXiv preprint: https://arxiv.org/abs/2003.12638

Winner of the "Best Student Paper" award. More info: https://vscl.tamu.edu/2020/06/15/vscl-alumni-vinicius-g-goecks-receives-best-student-paper-award/

Download our Dataset

Link to dataset: https://drive.google.com/file/d/1dDEGr9nHMK_3iXs0R1PSSIDqzOAxKqFa/view?usp=sharing

Thermal camera calibration files: https://drive.google.com/file/d/1SWOR8WUtou6KswemkB2yUqakHwq_jJFP/view?usp=sharing

Example MATLAB files on how to use the calibration files to combine RGB and LWIR frames (by Junyan Cao, NUAA): https://drive.google.com/file/d/1LsXMe78AxTvfO6UUnTVBMNRaCX93GszF/view?usp=sharing

The dataset contains images from three synchronized RGB, long-wave infrared (LWIR), and near-infrared (NIR) cameras. There is a folder for each data collection condition/day, and for each condition/day, there will be more three folders: NIR, RGB, and THRM. Inside each folder, there will be two separate ones with images and labels. Most of them have labels for both THRM and RGB at different frames.

For any questions, please contact Vinicius Goecks. Most up to date contact information can be found at https://vggoecks.com/.

Page updated

Google Sites

Report abuse