I am working with Ford Motor Company to develop the first commercially available autonomous vehicle to provide safe, affordable and ubiquitous mobility to all the consumers.  In particular we are exploring the possibility of using recently discovered deep learning techniques to solve some of the challenging problems faced in autonomous vehicles. It is not possible to pre-program the autonomous vehicle for all possible situations that may arise on the road. So autonomous vehicles will have to be intelligent enough to learn from what they observe, similar to the way we humans do. We will solve some of these challenging problems using fused sensor data and deep neural networks. We will use the fused multi-modal sensor data to develop an unsupervised learning algorithm using deep neural networks to detect and classify objects that an autonomous vehicle observes in an urban environment. Detection and classification of objects is very important for the successful operation of an autonomous vehicle. For example, it is important to distinguish if an object protruding from the ground plane on the pavement next to the car is a fire hydrant or a child, because the response of the vehicle may change based on this information. 

We have developed an information theoretic framework for multi-modal sensor data fusion for robust autonomous navigation of vehicles. In particular we have focused on the registra- tion of 3D lidar and camera data, which are commonly used perception sensors in mobile robotics. Our framework allows the fusion of the two modalities, and uses this fused information to enhance state-of-the-art registration algorithms used in robotics applications. The amount of information obtained from one sensor alone is not sufficient for complex tasks like place recognition, especially when the data in the map is significantly different from what the robot perceives in real- time. In situations where the robot operates in changing environments we either need more information, which can be obtained from multi-modal sensors mounted on the robot, or the prior map needs to be updated frequently enough so that the sensor data can be registered with data in the map. It is generally not practical to update maps of large environments so frequently. Therefore, the ability of our algorithm to fuse multi-modal data to recognize places across seasons, with significant appearance changes makes it very suitable for long-term autonomous operation of robots, without the need of updating the prior map.