Safe & Efficient Ground Exploration Using Deep Reinforcement Learning

In Fall 2021 I took ROB 537: Learning Based Control. This course focused on utilizing learning methods (neural networks, RL, evolutionary algorithms) to control robots. For our course project my team (Kyle Mathenia and Aiden Shaevitz) and I sought to create a simulated agent that was capable of high level navigation of rough terrain, trained on real world elevation data. The goal of this project was to train an agent that would be able to leverage local elevation readings to safely and efficiently aid in search and rescue missions. The culmination of this project was the writing of a conference style paper, for which we were awarded with the distinction of runner up for best paper of the class.

According to the US National Park services over 45000 people were lost and required search and rescue from 2004 - 2014. Drones are already being leveraged to great effect to aid in these operations but can be limited by weather and dense foliage. Ground based robots don't face these same challenges and are also able to deliver larger payloads such as critical first aid supplies. The main challenge for ground based robots is instead navigating harsh natural terrain.

Our environment was formulated as a classic grid world (just with very large grids) where the agent spawns in a random location and must reach a random target destination. The agent is a deep neural network that takes as input a small local viewing window of the surrounding elevation and a guiding compass unit vector that points towards the target. The output of the network is one of four actions representing the cardinal directions.

The elevation data utilized was collected from 0.5m LIDAR scans. The corresponding slopes of the terrain were converted into cost maps for each location with harsher penalties associated with crossing more treacherous terrain.

This project proved to be a difficult problem but very rewarding and informative. Our aim was to show that we could train an agent that was capable of making terrain aware planning decisions on a large space.

As a baseline we trained a more myopic agent with very dense rewards that was very good at getting to the goal but not very terrain aware. We compared this to our curriculum trained agent that saw sparser rewards and (we hoped) was more aware of its surroundings.