Results
3D Model Based Pose Estimation
Having a 3D model of a particular drone, we can design efficient algorithms do detect it and estimate its pose from a sequence of images.
We have created synthetic datasets of fixed wing drone flights in multiple realistic environments and arbitrary poses. This dataset was used to train and evaluate automatic pose estimation algorithms to track and command the motion of the drone from cameras aboard the ship.
This is the architecture of the neural network used to support the pose estimation algorithms. The network learns to remove the background from images with the drone. This is easy to do with the synthetic dataset because we can generate images with and without background of the drone in the exact same pose.
These are some results of the pose estimation algorithm. We show the ground truth pose of the drone in black and the estimated pose in red. Note the good overlap between the two, which indicates the correct operation of he method.
A requirement of the pose estimation method is that the input images are already cropped around the drone. Thus, an object detector is trained with a large dataset of real synthetic drone images with bounding box annotations.
Here are some annotations of our drone detector in real images.
Detection and Tracking of Objects in the Sea
Maritime and costal surveillance can be automated by image analysis. However, these are challenging environments due to sun glare, wave crests, wakes from boats, etc. We have developed several methods for the detection of ships and people in maritime environments exploiting machine learning models and custom built real and synthetic datasets.
We have created real and synthetic datasets of maritime environments with multiple realistic types of vessels and motions in arbitrary poses. This dataset was used to train and evaluate automatic algorithms to detect and track ships on the ocean from aerial image perspectives.
In this figure we can see the results of our detection networks in a challenging case of a ship barely visible due to sun glare and rough sea conditions. The image on the left is an image taken by a UAV flying over the ocean at about 300m. The image in the right shows the output of our detector network as a bounding box overlayed to the detected target.
In this figure we can see the results of our segmentation networks on single frames of ships capture by UAV's. We have proposed a processing pipeline composed of detection (YOLO), segmentation (U-NET), and refinement (CRF). WE can realize the advantaces of using CRF after the segmentation network, as the quality of the segmentation is of much better quality.
The figures above show the output of our segmentation models. A sequence of images taken by a UAV is processed sequentially and the corresponding segmentations obtained with temporal coherency.
Wildfire Monitoring
Wildfires are one of the biggest natural disasters that affect Portugal every year. They can cause extensive damage to property, natural resources, and wildlife, as well as pose a threat to human lives and communities. Aerial image processing can help on the early detection of fires as well on their mapping during combat to help decision making in the firefighting process. We have developed several methods to detect and geolocate fires in forests usign aerial images, both in the visible and the thermal spectrum.
One of the main requirements of any image based wildfire monitoring method is the ability to locate in the images fire and smoke regions. A challenge for creating such methods is the difficulty of annotating images for fire and smoke at a large scale. To address that problem we have proposed a weakly supervised method that used a combination of a image classifier, the class activation mapping method, and conditional random fields, to perform quite accurate segmentations of fire and smoke in images.
Thermal images are an important complement to RGB images. Thermal images are not very sensitive to smoke, so they can see the fire even in dense smoke conditions. However, they cannot distinguish so well the active flame from other very hot areas already burned. SO, we have developed methods with level set segmentation algorithms to segment the thermal images in different areas and locate the firefront.
After detecting the fire or smoke in images it is necessary to compute their geographical location in latitude / longitude coordinates, so the map of the situation can be updated with real time data.
We have developed methods that use both the UAV navigation sensors (GPS, IMU) and perform 3D reconstruction of the landscape with the captured images to match with the Digital Elevation Map. The combination of these methods (Direct and Indirect geo-referencing, improves the geo-localization of the events.
