Home

Real-time Forest Fire Detection System using Computer Vision and Deep Learning

Contributors

Nishant Bharali, Amal Sujith

Abstract

Global forest ecosystems, wildlife habitats, and human settlements are increasingly endangered by the phenomenon of forest fires. The early identification of incipient forest fires is a crucial element in preventing their uncontrolled proliferation over extensive geographical regions. Nevertheless, the present surveillance infrastructure for forest fire detection exhibits notable deficiencies. For instance, satellite-based systems such as MODIS are often impeded by their coarse spatial resolution and delayed response times, typically ranging between one to two days, as identified in the study by Li et al. (2018). Additionally, traditional methods such as watch towers and ground patrols are constrained by their limited area coverage, rendering vast forest areas unprotected. In response to these challenges, the current project is directed towards the development of a comprehensive, real-time forest fire detection system, leveraging the latest advancements in the realms of computer vision, deep learning, and unmanned aerial systems for continuous, extensive area monitoring. This research endeavors to create a highly accurate and efficient detection system utilizing advanced computer vision and deep learning methodologies, encompassing object detection and image classification. The core of this system is based on custom-developed machine learning algorithms that analyze collected data to ascertain the probability of imminent fire events, drawing upon recognized fire signatures. In parallel, aerial imagery obtained is subjected to analysis by deep convolutional neural networks, specifically tailored for fire detection. These networks adeptly categorize image segments into 'fire' or 'non-fire' classes, achieving an accuracy rate exceeding 80%. Current research efforts are concentrated on refining the model's accuracy, particularly in conditions of reduced visibility and over extended detection ranges. The proposed system represents a substantial advancement over traditional satellite-based and manual monitoring methods, which are limited in terms of responsiveness and coverage. With ongoing enhancements, this system has the potential to be scaled up for widespread application, offering a proactive solution to this pressing global issue.

Index Terms—Communication Bridge, Coverage Path Planning, Detection Algorithm, Fixed-Wing, Ground Control Station, Image Processing, Mission Planner, Navigation, Unmanned Aerial Vehicle, Convolutional Neural Networks

Introduction

The foundational motivation behind the conception and execution of this model was to extend support to the tribal communities who are heavily reliant on forest resources for their sustenance. Catastrophic events like forest fires have a devastating impact on these ecosystems, leading to the loss of both natural resources and wildlife habitats within these sanctuaries. Such incidents disrupt the ecological balance in these regions significantly.

In this context, the capacity for real-time detection emerges as a crucial tool for forest management authorities, offering immediate insights into unfolding situations. Traditional satellite-based detection methods are becoming increasingly obsolete due to their prolonged relay times, which hampers timely response to such emergencies. In contrast, the utilization of drones emerges as a superior alternative, spearheading the detection of these perilous events.

Furthermore, the ethos promoted by our university, which emphasizes compassion towards those in need and the importance of contributing positively to society, was a critical factor in our decision to undertake this project. This initiative aligns with our institution's commitment to fostering social responsibility and aiding communities that depend on natural resources for their livelihood.

Figure 1. Proposed Workflow

Approach

This study will utilize advanced Convolutional Neural Network (CNN) architectures, with a focus on integrating Faster R-CNN Inception models into Raspberry Pi 4 systems for the purpose of forest fire detection. The training of these models will involve a dataset comprising images of forest fires, sourced from the National Disaster Management Authority of India's (NDMA) online repository, alongside images depicting standard forest landscapes without fire presence. To enhance the efficacy of these models, we will implement transfer learning techniques, leveraging the capabilities of pre-existing neural networks such as Inception and MobileNet. This approach involves the modification and fine-tuning of both the classifier and regressor components within these networks, specifically tailoring them for fire detection tasks. Additionally, data augmentation strategies—including image flipping, rotation, and adjustments in brightness and contrast—will be employed to broaden the variability within the training dataset.

Our model development and testing phases will be conducted on a high-capacity NVIDIA GPU server, ensuring accelerated computational processing. Following the determination of the most effective model, we will proceed with its conversion and subsequent deployment on compact computing devices, such as the Jetson Nano and Raspberry Pi. This enables real-time fire detection capabilities when integrated with aerial vehicles like fixed-wing gliders or Tilt Rotor VTOLs in a controlled simulation environment. The deployed model will be responsible for processing live video feeds from mounted cameras on these vehicles, identifying and delineating fire-affected areas through generated bounding boxes.

For a comprehensive implementation, our project is structured into several distinct phases.

The Coverage Strategy we are employing:

Development of a Custom Python Script Utilizing DRONEKIT for Indigenous Control.
Pre-setting Flight Simulation Parameters.
Simulation of Fixed-Wing Aerial Vehicles.
Implementation of Fire Detection through Advanced Computer Vision Techniques.
Enhancement of Predictive Accuracy of the Model.

Code Implementation:

I. Proposed Methodology for UAV Based Object Detection

The proposed model, illustrated in Figure 2 below, consists of two primary components: an aerial forest fire detection system using Unmanned Aerial Vehicles (UAVs) and a Wireless Sensor Network (WSN) based system for data collection and future temperature spike prediction.
The aerial detection system, employs a remote-controlled, fixed-wing UAV (shown in Figure 2b) equipped with a high-resolution camera, an Ardupilot-based autopilot Pixhawk module, a Raspberry Pi microprocessor, and other essential electronics. A deep learning model, trained on images from artificial forest fire scenarios and annotated with 'fire' labels, was developed on Google Cloud and optimized into a TensorFlow Lite model for efficient operation on the Raspberry Pi. The UAV's navigation and guidance are managed by the Pixhawk module, and it communicates fire detection alerts and GPS coordinates to the ground station using a Seed Grove Lora-WAN module at 865MHz.
This application employs Multiple Linear Regression (MLR) to predict future temperatures, corroborated with an average temperature generating model for enhanced accuracy. Together, these systems provide round-the-clock surveillance with an accuracy exceeding 75%.

Figure 2. Expected Flight Operation (UAV based implementation)

The Figure indicates the work cycle of the drone with logic block and detection condition. The ultimate goal is to detect with maximum accuracy during the mission and when faced any critical battery failure or draining, then to safely return back to the base with minimal damages. This workflow provides an idea that has been presented in paper [13], where the Swarm drones return back to the base in case of repairs and is predominantly used for Autonomous flights along with manual flight control capability for human intervention.

Deviations:

As causing a large-scale fire is not plausible in a real-case scenario to test the system, we are preferring simulations to replicate the same and test the procedure for small area-of-effect scenarios in the real world.

Required Tools

Fixed Wing Type of Drone is Utilised for Detection of Forest Fires.
PIXHAWK FLIGHT CONTROLLER is the Brain of the UAV.
It consists of in-built 3-Axis Gyro meter, Accelerometer, High-Performance Barometer and Magnetometer.
It works on 32-bit Computer where the sensors can be calibrated and autonomous missions can be planned with help of GPS Module.
Previous Work included Simulation of Flight Plan in Mission Planner

The existing algorithm is based on the HSV based detection based on OpenCV will be upgraded to a deep learning-based model.
The new model can give better performance even in different lighting conditions by using a variety of training set.
The trial run for the collection of the dataset is completed.
Both image classifier and detector can be developed using the dataset and TensorFlow environment.
Tkinter and OpenCV based user interface can also be developed for better user experience.
RADIO TELEMETRY - 3DR Single TTL Mini Radio Telemetry is currently employed in the Project.
It has Two Modules i.e., Air and Ground which is used in Transmission of Data from Glider. It can use UART and USB Connections.
ELECTRONIC SPEED CONTROL(ESC) - It is a electronic circuit that regulates the Speed of the Electric motor.
GPS MODULE - It comes with in-built digital compass and rechargeable battery for warm starts.
It has an accuracy upto 0.6 to 0.9 meters
ELECTRIC MOTOR - Avionic Skysurfer V4 brushless motor. Can generate up to Maximum Power of 230 Watts
Lipo Battery 3 cell, 11.1V, 40c

Working

The glider consists of two cameras, Video telemetry unit, GPS module, Smoke sensor (MQ2) and Micro-controller (Raspberry Pi 3B+/ NVIDEA Jetson Nano) etc. Tensor Flow software is installed in the micro-controller which uses RCNN machine learning algorithm for Fire detection. RCNN is trained with 3000-4000 images of fire in different scenarios. If normal fire image is detected the communication part is made active but if a smoky atmosphere the sensor part of the glider is made active to check the amount of carbon dioxide present in the atmosphere. Readings in smoke sensor (MQ2) and if the reading is above a particular threshold level then the communication part is made active. The other camera of the drone gives the live streaming of the location to the forest officials to avoid false alarm.

II. Proposed Idea for Ground Based Detection Module using WSN

A cluster of LM35 (Temperature Sensor) is used to detect temperature.
To increase the sensitivity the sensor network is Integrated on aluminium rod which have high heat conductivity compared to ground.
The Routers contain Lora Wan module which communicate with the nearby routers as well as the central coordinator to activate the alert signal by enabling GSM.
After consecutive testing for 3 months the estimated accuracy is about 85%+.
The module is powered with lithium-ion battery which is charged using renewable source of energy (i.e. using solar power).

Working

The algorithm works by storing the sensor value in an array of 12, by collecting the temperature value in 2 mins (i.e. 12 reading in 2 mins) with 10sec gap in consecutive reading.
Average of temperature is taken in every 2 mins.
If the average is above the set threshold value the alert 1 is counted and communicated to central module.
If the average again crosses threshold, router interpret as hike in temperature and central coordinator is awaken to send an alert message to the officials.

Implementation Images

Experiments and Results

1. Fire Detection Using Deep Neural Networks

The study developed custom deep neural networks to identify fire scenes by analyzing discriminative features such as color, shape, and texture patterns. A Faster R-CNN model with a ResNet-50 backend was identified as more accurate over SSD alternatives from precision-recall analysis, despite its higher computational requirements. For efficient deployment on Raspberry Pi 4 target inference platform, models were converted to TensorRT format.

1.1. Forest Fire Detection Algorithm

A systematic two phase methodology was adopted for developing and optimizing a vision-based deep neural network model for reliable wildfire identification capability from imagery under challenging real-world conditions.

Phase 1: Rule-Based Method

As an initial baseline technique, a color thresholding based model was implemented leveraging OpenCV for fire pixel segmentation within images.
The key steps are:

Capture video frame from embedded camera at 30 FPS.
Convert RGB encoded image to HSV color space HSV allows isolating luminance and chrominance components allowing better adaptation across lighting variations.
Define lower and upper thresholds for fire pixel hue, saturation and value ranges.
Apply threshold to filter image retaining only fire colored pixels, rest set to black.
Perform logical AND operation between filtered binary mask and original frame.
Count number of non-zero bits to detect presence of fire colored regions.
Trigger alert status if fire pixel count exceeds minimum threshold.

This intuitive and computationally inexpensive approach serves as a fast method for fire identification. However, limitations were noted regarding poor tolerance to noise, shadows and outdoor luminance variation deteriorating detection accuracy.

Phase 2: Deep Learning Model

To overcome limitations of color rules-based method, a more robust deep convolutional neural network architecture customized for fire recognition was developed leveraging transfer learning:

Curate wildfire image dataset from multiple sources documented in the Dataset Formulation section.
Manual annotation of images for classification and localization objectives.
Design model leveraging deep CNN advances - Faster R-CNN and SSD variants.
Initialize models with ImageNet pre-trained weights for effective generalization.
Augment data applying random transformations enhancing appearance diversity.
Optimize model hyperparameters maximizing accuracy metrics over multiple iterative experiments Batch size, learning rate, epochs etc.
Convert optimized model to TensorRT format for maximizing inference performance on embedded Jetson platform.
Deploy model for real-time analysis of 1080p aerial video streams during field trials.

Among the models, SSD framework offered optimal accuracy-latency trade-off by simplifying the detection to a single shot prediction network compared to the two-stage region proposal mechanism of Faster R-CNN derivatives. The final model operates at over 80% precision in identifying diverse fire events within aerial imagery while maintaining 12 FPS throughput on the Jetson module. Table I summarizes the frame-by-frame analysis rate and accuracy achieved using the DNN models on the target embedded hardware alongside the software-based rule technique. Fig. 1 and 2 showcase sample detections using the deep network architecture.

1.2. Comparative Analysis

Table I. Performance of Experimental Fire Detection Models

Figure 3.1. SSD mobilenet v2 model

Figure 3.2. Faster RCNN inception v2 Model

A comparative analysis of the models was conducted, demonstrating their frame rate and accuracy in fire detection. The Faster R-CNN model had a higher accuracy (86%) but a lower frame rate (10 FPS), whereas the SSD-MobilenetV2 achieved 83% accuracy at a slightly higher frame rate (12 FPS). The rule-based method, while faster (15 FPS), lagged in accuracy at 62%.

By harnessing deep neural network architectures combined with well annotated datasets and model optimization strategies, the solution demonstrates reliable automated fire spotting capability from sky allowing early alerts to ground response teams monitoring expansive territories.

Figure 3.3. Pre-processed Image and Original Image

Figure 3.4. Live detection using Faster RCNN inception v2 - Test Case 1

Figure 3.5. Maximum FOV for Test Case 1

Figure 3.6. Live fire detection test for a distinct scenario

1.3. Detection Through Computer Vision

In the detection part, the sample fire images are captured at the rate of 30 fps, using a normal camera and the captured frames are converted from the RGB color code to HSV format for the better detection. Here HSV is having the upper hand because HSV can separate the lumina component of the frame from the chroma component, this property of HSV will give better output in different lighting conditions. In order to set up a filter, masking is done in a particular range of color spectrum including the combination of yellow and red. Later this mask and the real image is passed to a filter which will perform a bit-wise AND operation. The output of AND operation in the black/white domain:

Filter response:

“Color in the range from frame” AND “color in the range from mask” = same color
Same color AND black = black

This operation will give the final filer output, later a non-zero bit counter is used on the filter response to produce the final prediction of fire.

1.4. Comparative Results

We further performed tests on both the models and gathered the following results for both the models and performed a comparative analysis between them:

The model was able to provide accuracy above 80% on ideal conditions with a minimum of 60%.

Detection algorithm based on faster RCNN inception V2 gave higher accuracy and Higher processing time.
SSD Mobilenet V2 gave comparatively lower accuracy but processing time was better.
Model is converted to tflite format for edge applications.
A OpenCV Based python file was developed for connecting and providing a meaningful front end for the detection.

Qualitative Results

Challenges faced and success/failure cases

Live detection using SSD Mobilenet v2 model was giving varying results, resulting in a few failures in the live detection of the fire compared to the other model which gave consistent results. The failure frequency was low to consider any impact in the implementation and results.
We felt the need for an external GPU to be integrated with Raspberry Pi to improve the performance of detection as we faced issues leading to lag, slow processing time and crashing of the models sometimes.

Fire Detection success message and Output Image verification is presented as follows:

Figure 3.7. Indicates the Successful fire detection and Output Image

Video footage of test cases:

Real-time Fire Detection using Faster RCNN inception v2 model Test Case 1 Footage:

Untitled video - Made with Clipchamp (4).mp4

Real-time fire detection for Faster RCNN inception v2 model - Test Case 2 Footage:

Untitled video - Made with Clipchamp (5).mp4

Additional Footage of UAV based Implementation:

CV UAV.mp4

Conclusion

Recent research proposes an integrated system for forest fire detection that appears to offer faster and more reliable outcomes compared to existing methods. The use of unmanned aerial vehicles (UAVs) enables extensive monitoring of hard-to-reach forest areas, while a wireless sensor network provides complementary detection capacity when drones are grounded. Preliminary tests suggest the combined system can reduce false alarms through cross-validation. Additionally, refinements to the image processing algorithms show promise for detecting both smoke and flames in challenging real-world conditions. While early results are encouraging, the persistent issue of high false alarm rates indicates that further research is needed to realize the full potential of this integrated approach. With additional training data and algorithm refinement, there may be opportunities to significantly improve accuracy. By addressing existing limitations, next-generation versions of the system could become a versatile component of forest fire prevention and response.

Future Scope

Conditions to find that provide viable alternatives for reduced power consumption:

To look for an efficient AI based model that can consider number of factors for better prediction under diverse conditions.
To increase the accuracy of predication from higher altitude since realistically the live footage will be taken by glider which fly in high altitudes in an official scenario.
To achieve a minimum of 95% precision and recall on the test set and raise the accuracy for at least one of the models to 90% and above with minimal failures, i.e. missing fire detection and false alarms.

References

Guruh Fajar Shidikl, Fajrian Nur Adnan, Catur Supriyanto, Ricardus Anggi Pramunendar, Pulung Nurtantio Andono: "Multi-Color Feature, Background Subtraction, and Time Frame Selection for Fire Detection", 2013 International Conference on Robotics, Biomimetics, Intelligent Computational Systems. DOI: 10.1109/ROBIONETICS.2013.6743589
F. Valenti, D. Giaquinto, L. Musto, A. Zinelli, M. Bertozzi, A. Broggi: "Enabling Computer Vision-Based Autonomous Navigation for Unmanned Aerial Vehicles in Cluttered GPS-Denied Environments," 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, 2018, pp. 3886-3891. DOI: 10.1109/ITSC.2018.8569695
B. C. Ko, S. J. Ham, J. Y. Nam: “Modeling and formalization of fuzzy finite automata for detection of irregular fire flames,” IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 12, Dec. 2011.
S. J. Pan and Q. Yang: “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, Oct. 2010.
K. Muhammad, M. Sajjad, M. Y. Lee, S. W. Baik: “Efficient visual attention driven framework for key frames extraction from hysteroscopy videos,” Biomed. Signal Process. Control, vol. 33, Mar. 2017.
A. Ullah, J. Ahmad, K. Muhammad, M. Sajjad, S. W. Baik: “Action recognition in video sequences using deep Bi-directional LSTM with CNN features,” IEEE Access, vol. 6, 2017.
A. Ullah et al.: “Action recognition in movie scenes using deep features of keyframes,” J. Korean Inst. Next Generation Comput., vol. 13, 2017.
A. Viseras, J. Marchal, M. Schaab, J. Pages, L. Estivill: "Wildfire Monitoring and Hotspots Detection with Aerial Robots: Measurement Campaign and First Results," 2019 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Wurzburg, Germany, 2019, pp. 102-103. doi: 10.1109/SSRR.2019.8848961.
O. Surinta and S. Khruahong: "Tracking People and Objects with an Autonomous Unmanned Aerial Vehicle using Face and Color Detection," 2019 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT-NCON), Nan, Thailand, 2019, pp. 206-210. doi: 10.1109/ECTI-NCON.2019.8692269.
P. Foggia, A. Saggese, M. Vento: "Real-time fire detection for video surveillance applications using a combination of experts based on color, shape, and motion," IEEE Trans. Circuits Syst. Video Technol., vol. 25, no. 9, Sep. 2015.
V. Zope, T. Dadlani, A. Matai, P. Tembhurnikar, R. Kalani: "IoT Sensor and Deep Neural Network-based Wildfire Prediction System," 2020 4th International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India, 2020, pp. 205-208. DOI: 10.1109/ICICCS48265.2020.9120949.
C. Gomathi1, K. Vennila: "Wireless Sensor Network for Forest Fire Detection and Decision Making," International Journal of Advances in Engineering Science and Technology, Nov. 2013. DOI: 10.1109/ICIEA.2009.5138270.
Taua M. Cabreira, Lisanne B. Brisolara, Ferreira Paulo R. Jr.: "Survey on Coverage Path Planning with Unmanned Aerial Vehicles," Drones Journal MDPI, January 2019. DOI: 10.3390/drones3010004.
Hasan M. Qays, Dr. Bassam A. Juma, Dr. Ayman D. Salman: "Design and Implementation of Autonomous Quad-copter using SITL Simulator," Iraqi Journal of Computers, Communications and Control Systems Engineering (IJCCCE), January 2020. DOI: 10.33103/uot.ijccce.20.1.1.
Erich Seifert, Stefan Seifert, Holger Vogt, David Drew, Jan van Aardt, Anton Kunneke, Thomas Seifert: "Multi-View Reconstruction of Forest Images," Journal on Remote Sensing MDPI, May 2019. DOI: 10.3390/rs11101252.
Kevin Sheng-Wei Lim: "Lidar remote sensing of forest canopy and stand structure," Thesis Paper, Queen’s University, Ontario, Canada, May 2006. ISBN 2527-6638.

Page updated