Embedded Files
USE CASE: INVASIVE SPECIES DETECTION
"Land managers need to know where invasive species are established on the landscape to be able to undertake measures to kill or remove them, but the vast expanses of potentially infested territory preclude repeated, comprehensive field surveys." This is where robotic surveillance and mapping systems could prove useful.
https://eros.usgs.gov/doi-remote-sensing-activities/2016/usgs/detection-invasive-plant-populations
ENVIRONMENTAL SURVEILLANCE AND MAPPING
ENVIRONMENTAL SURVEILLANCE AND MAPPING
I am interested in using Artificial Intelligence and Machine Learning to solve environmental problems. An important aspect of being able to do so is to first be able to examine and measure environments. At present, the only well documented methods for doing wide-area surveillance and detection of objects are LiDAR-based mapping solutions. So, I set out to use LiDAR to map the St. Stephens' campus and then use the map to solve an environmental problem, specifically to detect invasive plant species on campus. There are two significant hurdles:
a) LiDAR is expensive, and most solutions on the market cost upwards of $20,000! So how can I make a $20,000 solution viable at ~$1,000?
b) LiDAR maps are easy to generate when the sensor is static. However, when the sensor itself is moving (as it is in remote mapping situations) the laser points received must be registered in some coordinate system. This is the problem of SLAM (Simultaneous localization and mapping - see below). One solution is to use odometry or motion sensors to determine the robot's change in position relative to some known position. This is a difficult problem that is explained here and one I am trying to solve using a method called LOAM.
"The Simultaneous Localization and Mapping (SLAM) problem asks if it is possible for a mobile robot to be placed at an unknown location in an unknown environment and for the robot to incrementally build a consistent map of this environment while simultaneously determining its location within this map. A solution to the SLAM problem has been seen as a ‘holy grail’ for the mobile robotics community as it would provide the means to make a robot truly autonomous. " Durrant-Whyte and Bailey
Also learn more from: Arshad, S.; Kim, G.-W. Role of Deep Learning in Loop Closure Detection for Visual and Lidar SLAM: A Survey. Sensors 2021, 21, 1243. https://doi.org/10.3390/s21041243
WHY LIDAR?
WHY LIDAR?
LiDAR or Light Detection and Ranging, is a remote sensing method that uses light in the form of a pulsed laser to measure ranges (variable distances) to the Earth. These light pulses—combined with other data recorded— generate precise, three-dimensional information about the shape of the Earth and its surface characteristics. A significant advantage of Lidar is that it is not sensitive to ambient lighting or "optical texture" making it very suitable for all conditions.
A LiDAR instrument principally consists of a laser, a scanner, and a specialized GPS receiver. Typically, airplanes and helicopters are used for acquiring lidar data over broad areas. Two types of lidar are topographic and bathymetric. Topographic lidar typically uses a near-infrared laser to map the land, while bathymetric lidar uses water-penetrating green light to also measure seafloor and riverbed elevations.
LiDAR systems allow scientists and mapping professionals to examine both natural and manmade environments with accuracy, precision, and flexibility.
Source: https://oceanservice.noaa.gov/facts/lidar.html
The PROPOSED SOLUTION
The PROPOSED SOLUTION
Project Goals: These are not final and will likely change based on the feasibility of completing them within a 5-week timeframe
Identify a cost-efficient LiDAR sensing system
Mount the LiDAR sensor on a mobile robot to map various parts of campus (this is part of my Projects in Science and Technology class with Mr. Lanier)
Acquire data using LiDAR (Livox MID-40 sensor)
Create a 3D map using Lidar Odometry And Mapping (LOAM)
Use a second layer of machine learning code/models that can identify objects (invasive species) using the 3D map. This component will be part of my continuing research this summer because it is just not viable to complete with the time I have.
Team Members
Team Members
SSES student, Project Leader
UT Austin CAM Chair in Visualization
SSES, Robotics and Engineering Faculty
Nathan Elias
LASA student, Project Team Member
Week 1: 04/06/22- 04/13/22
I started this week by meeting with my project collaborator, Nathan Elias, and we discussed the progress we have made since we last met. He had made progress on the second layer of Machine Learning code that will eventually be able to identify objects from the 3D LiDAR map. I had made progress in the implementation of the LOAM code for map generation. I was originally using the official Livox implementation of LOAM, but decided to switch to an implementation by HKU-Mars because it had a more active Github repository (more questions being asked and resolved in the "issues" tab) and seemed to be more reliable than the official version. I also found one issue in the HKU-Mars repository that had exactly the same setup that I had and the exact same issue that I had (a very rare occurrence in coding). This person was able to fix this issue by changing the version of Linux on which they were operating and switching from ROS Melodic to ROS Kinetic. This is what I am going to try over Easter weekend, and once I do so, I will have a much clearer understanding of the feasibility of my goals moving forward.
I also met with Dr. Chandrajit Bajaj (my external mentor and the CAM chair of Visualization at UT Austin's Oden Institute). He gave me some insight into a possible Inertial Measurement Unit (IMU) that I could attach to my LiDAR sensor. This was amazing because I previously thought that IMUs cost well over $1000, so I will research methods for integrating it into the system. (Note: For airborne LiDAR, IMUs track tilt. LiDAR systems use tilt to accurately measure the incident angle of the pulse.)
A significant problem with using odometry for registering lidar laser points is that as odometry uses small, incremental movement over time, it is bound to compound error and thereby create what is known as drift (leading to both poor localization as well as difficultly in linking different maps). To date, offline 3D maps have been built using loop closure to correct for drift over time. Ji Zhang and Sanjiv Singh's method (LOAM) achieves both low-drift and low-computational complexity. According to Zhang and Singh, key to this level of performance is the division of the complex problem of simultaneous localization and mapping (SLAM) by two algorithms. One algorithm performs odometry at a high frequency but low fidelity to estimate velocity of the LiDAR. The other algorithm runs at a frequency of an order of magnitude lower for fine matching and registration of the point cloud. By combining the two algorithms, LOAM is capable of mapping in real-time. LOAM has been thoroughly tested and evaluated and the results indicate that it can achieve comparable accuracy to state of the art offline batch methods without needing expensive add-ons.
Week 2: 04/14/22 - 04/21/22
Over Easter weekend (4/15 - 4/18), I was able to convert unprocessed LiDAR data into a map for the first time using a LOAM model!! The output map that I created was the exact same as the result from the HKU-MaRS repository (right). This is a big moment for me.
However, this was done using data that had already been acquired, so the map wasn't generated in real-time. I spent 4/19 - 4/21 modifying my code to make a map in real-time, but I still haven't figured out the source of the problem. Based on the ROS display, the sensor seems to be sending the data to my PC without any issues. However, it is not doing any of the odometry or pose estimation that it had done when mapping the sample data. My next steps will be to sift through the "issues" section of the HKU-MaRS repository to see if anybody else has had a similar issue. My current hunch is that my code is getting overwhelmed by the amount of data it's being given by the sensor because its own output is much slower than the sensor. This will create a build up of data which causes the code to automatically shut down the entire process. I will try to debug this over the weekend and provide an update with the results.
Output map using the HKU-Mars sample LiDAR data.
Colors measure intensity on a scale of 0 (blue) to 255 (red). Intensity is a measure, collected for every point, of the return strength of the laser pulse that generated the point. It is based, in part, on the reflectivity of the object struck by the laser pulse (other factors that impact intensity include incident angle, range, surface composition etc.). Light intensity is useful in distinguishing features in land use/cover. For example, impervious surfaces stand out in light intensity images. This is why light intensity is good for image classification like object-based image analysis.
Source: https://gisgeography.com/lidar-light-detection-and-ranging/
Trouble-shooting
On the right is a split-screen view of my code (left) and the ROS display (right). When I turn on the sensor and it starts sending data, the terminal on the left sends an error message within 1 second saying "process has died... killing on exit." This basically means that it got overloaded by something, so it shut itself down.
Week 3: 04/22/22 - 04/28/22
Over the weekend (4/23 - 4/24), I met with Dr. Bajaj and Nathan to discuss next steps on the project. A screenshot of these goals and tasks can be seen in the image to the left, but their contents are more technically involved because it involves researching the benefits and drawbacks of different ML models for object (invasive species) detection. This research is "Step 1" in the images and some of the work that I have been doing and the papers I have been reading are also referenced there.
I also continued my testing with the MID-40 sensor and was able to generate a map of a room in my house. Through this experimentation, I learned a couple of things. First of all, the sensor doesn't do well in small areas. It doesn't detect objects closer than 6-7 ft. with any predictability or accuracy. Secondly, it doesn't like reflective, transparent, or translucent surfaces (especially glass), because they create noisy data. As a result, detection using LOAM becomes much more difficult, because the noise in the data confuses the model and makes it think that the sensor is suddenly in a different position. What I concluded from my tests was that utilizing an IMU to get accurate positional and directional data will be very helpful in maximizing accuracy and confidence of the map while also minimizing the impact of noisy data.
On Wednesday 4/27, I tried mapping Mr. Lanier's room (Lab 16) with the sensor, but the output was completely messy and unreadable. I believe that this is due to the amount of glass and other shiny things that create noisy data. I will be gone for SPC from 4/28 - 4/30, so I will try to create a map in an outdoor environment on 5/1. The greenery of nature should be much less reflective than anything indoors, so let's see what happens.
Advice to future seniors: I liked this project and thought it was beneficial because it gave me more time to work on a project that I had been exploring previously. However, I wish the SIP was for a longer duration. The small amount of time given to work on this project makes a truly meaningful project at times difficult. I would definitely recommend some pre-work so that you can use the SIP 5/6-week duration to make real progress.
Source: https://github.com/ghimiredhikura/Complex-YOLOv3
Shown above is a camera view and a LiDAR view of traffic from the perspective of a self-driving car. A YOLOv3 algorithm has been applied on the LiDAR point cloud data.
Note: YOLOv3 uses convolutional neural networks to identify objects in images. The network splits the image into regions (grids) and predicts object boundaries or bounding boxes using probabilities for each region. The same probabilistic model is used to predict the class of an object and then identify the object.
Week 4: 04/29/22 - 05/05/22
As I previously mentioned, I was out of town for SPC until 5/1. On 5/1, I learned some important things about my data acquisition and filtering. The Rosbag files that were used as examples on the github repositories I was working with had one "topic" which was sending messages. This topic was titled "/livox/lidar." I then checked the topics of the Rosbag files that my MID-40 sensor was creating, and sure enough, it was publishing 10 different topics!!! So it's not surprising that the GUI was getting overloaded with data because it was showing 10x more points than it should have been showing. I also tested whether changing the movement speed of the sensor would give better results than keeping it stationary and changing its orientation. I concluded that a moving sensor provides a better point cloud as the odometry and mapping algorithms are integrated.
During the week (5/2 - 5/5), I also had 2 AP exams so made less progress than I would have liked. I did research on comparing the accuracy, speed of algorithms, and methodologies of 3 object detection algorithms - DBSCA, YOLOv3, and PointNet. I decided that YOLOv3 was the most promising algorithm for high performance object detection on LiDAR data. (Accuracy is better than PointNet and comparable to DBSCAN; speed and efficiency better than both the other 2 and YOLOv3 also has more implementations available).
I also started working on my presentation for the SIP on 5/13.
Page updated
Report abuse