Combining Visible and Infrared Spectrum Imagery using Machine Learning for Small Unmanned Aerial System Detection
This research work proposes combining the advantages of the long-wave infrared (LWIR) and visible spectrum sensors using machine learning for vision-based detection of small unmanned air systems (sUAS). Utilizing the heightened background contrast from the LWIR sensor combined and synchronized with the relatively increased resolution of the visible spectrum sensor, a deep learning model was trained to detect the sUAS through previously difficult environments. More specifically, the approach demonstrated effective detection of multiple sUAS flying above and below the treeline, in the presence of birds and glare from the sun. With a network of these small and affordable sensors, one can accurately estimate the 3D position of the sUAS, which could then be used for elimination or further localization from more narrow sensors, like a fire-control radar (FCR).
Sensors
Summary Video
Complete Detection Videos
Comparison between LWIR+RGB, LWIR, and RGB predictions - single drone case - confidence 50%
Comparison between LWIR+RGB, LWIR, and RGB predictions - multiple drones case - confidence 50%
Publications
Paper published at The 2020 SPIE Defense + Commercial Sensing: Automatic Target Recognition XXX: https://spie.org/Publications/Proceedings/Paper/10.1117/12.2557442
arXiv preprint: https://arxiv.org/abs/2003.12638
Winner of the "Best Student Paper" award. More info: https://vscl.tamu.edu/2020/06/15/vscl-alumni-vinicius-g-goecks-receives-best-student-paper-award/
Download our Dataset
Link to dataset: https://drive.google.com/file/d/1dDEGr9nHMK_3iXs0R1PSSIDqzOAxKqFa/view?usp=sharing
Thermal camera calibration files: https://drive.google.com/file/d/1SWOR8WUtou6KswemkB2yUqakHwq_jJFP/view?usp=sharing
Example MATLAB files on how to use the calibration files to combine RGB and LWIR frames (by Junyan Cao, NUAA): https://drive.google.com/file/d/1LsXMe78AxTvfO6UUnTVBMNRaCX93GszF/view?usp=sharing
The dataset contains images from three synchronized RGB, long-wave infrared (LWIR), and near-infrared (NIR) cameras. There is a folder for each data collection condition/day, and for each condition/day, there will be more three folders: NIR, RGB, and THRM. Inside each folder, there will be two separate ones with images and labels. Most of them have labels for both THRM and RGB at different frames.
For any questions, please contact Vinicius Goecks. Most up to date contact information can be found at https://vggoecks.com/.