REU Project: Abhi Kamboj

The VREP pioneer robot and a 2D laser scanner was programmed to navigate through an unknown environment and to a desired location while avoiding obstacles. The algorithm was written in C++ and used ROS to communicate to the robot. This was done by generating a million potential trajectories and choosing the one which would bring it closest to it's goal while still avoiding obstacles. In addition, an emergency break was implemented in case the robot came to close to an obstacle (this usually occurs if it chooses a potential trajectory outside the range of its sensor). Below is an outline of the process, and images of the environments in which the robot was tested.

The same process was repeated as above, except this time the algorithm stored previous scans of the environment in a global point cloud array. Each of the scans were added to each other in relation to some fixed point on the map, and, as a result, the robot formed a local map of the areas it had navigated through. This modification drastically minimizes the error of running into objects that the sensor cannot see in one scan, because now the optimal trajectories are chosen with much more information of the surroundings. Below are images of the resulting local maps produced.

1. Obstacle Avoidance and Navigation

The algorithms described above (producing random velocity vectors and choosing an optimal one) were implemented on the Create robot, using the create_autonomy package as a low level controller, and using the urg_node package to publish scan messages from the Hokuyo sensor . After this portion of the project, the robot could autonomously navigate from one point to the next without colliding into obstacles.

After the navigation application was functional it was condensed into 2 launch files that ran the appropriate nodes:

• mapping.launch allows you to map the area
• navigation.launch will then allow you to navigate the mapped area.

Nowadays, many traditional control problems that require a large amount of computations are being replaced by reinforcement learning techniques. This summer I read much literature on the basics of reinforcement learning (RL) and wanted to include RL in the autonomous navigation techniques I was working on.

This project replicates a simple gridworld reinforcement learning problem in VREP, using the Quadcopter as an agent and trees as obstacles (the problem was still 2D). An action was a .25 meter step in the positive or negative x or y directions, and the state was observed every step of the simulation to be the current x,y position of the agent. The agent began at 0,0 and the goal was to reach 4,4. The agent was restrained from stepping to an x or y greater than 4 or less than 0; such an action simply causes the agent to stay in it's current position.

Rewards were distributed as follows:

• Large positive reward given at goal
• Negative reward given at collision (and simulation stops at collision)
• Positive reward inversely proportional to the distance from the goal given at every step
• Small negative reward for taking an action that would result in being out of bounds (x or y >4 or <0

Lua scripts were written to remotely control the VREP simulation with ROS. An epsilon greedy Q-Learning algorithm was implemented with a Q table in C++, and the Q table was stored in an external file.

After adjusting the hyperparamters to optimize the learning process, the problem was solved after 2000 episodes. A gif of the agent's navigation after 2000 episodes is shown below.

More detail can be found in my GoogleDoc linked below.

Code can be found on GitHub: https://github.com/akamboj2/AutonomousNav-RL_Research2018

Please contact me if you have any questions or comments: Abhi Kamboj (akamboj2@illinois.edu)