AI in Robotics

Machine Learning (ML) has helped revolutionize the field of robotics, particularly in the areas of computer vision, planning, control and sensor fusion. Our lab is currently investigating the following research topics within these domains:

Virtual Reality (VR) Teleoperation: Learning from human expert demonstrations is a key area of robotics. However, it can be difficult for a human to learn to control high degree-of-freedom robotic systems, making collecting these demonstrations very time consuming. VR teleoperation systems make demonstration collection fast and intuitive.
Learning from Demonstrations:
Fluid Handling: The task of interacting with and pouring transparent fluids is difficult for a few reasons. Firstly, detecting and tracking transparent fluids and vessels is a challenging computer vision problem. Secondly, models to predict fluid behavior must be real-time to be included within the robotic control loop. Our lab is tackling both of these problems to develop a robotic system to pour the perfect cup of soda.
Deformable Object Manipulation: Deformable objects are very prevalent in the world, yet they are incredibly challenging for robots to manipulate. This is because unlike rigid objects, they do not necessarily have any inherent structure, requiring a full 3D view of the object. Additionally, it is difficult to predict how the object structure will change based on interactions. Our lab is developing a robotic sculpting system to investigate the challenges of handling 3D deformable objects.
Autonomous Cooking: Autonomous cooking is a challenging domain within robotics that allows us to explore problems related to long-horizon task planning and completion, object detection and tracking of objects undergoing visual changes in cluttered environments, and the complex dynamics of interacting with deformable food.
Tactile Sensor Design: ML techniques allow for robotic control systems that can leverage multimodal sensor information. Our lab is actively designing a range of tactile sensors to provide our robotic systems with high-quality real-time feedback to improve performance, and generalizability.

Our Robotic Research at MAIL

VR Teleoperation

Our lab developed a VR teleoperation system that can collect high-quality human demonstrations, while remaining very intuitive to control for the user.

a) Minimizing Human Assistance: Augmenting a Single Demonstration for Deep Reinforcement Learning

Fluid Handling

Within the domain of fluid handling we are (1) creating a vision system to segment transparent liquids and vessels, (2) modeling fluid behavior and foam formation and integrating these models within the control loop, and (3) predicting fluid properties through interaction.

Papers: https://arxiv.org/pdf/2309.08892.pdf, https://arxiv.org/pdf/2308.02715.pdf

Project Website: https://sites.google.com/andrew.cmu.edu/robotcokepouring/

Deformable Object

Robotic systems handling deformable objects require full knowledge of the object’s shape. To that end, we developed a real-time 3D reconstruction pipeline to track the deformation of clay. Furthermore, we developed SculptBot, an autonomous sculpting robot that learns the underlying dynamics of clay deformation given an action.

Paper: https://arxiv.org/pdf/2309.08728.pdf

Project Website: https://sites.google.com/andrew.cmu.edu/sculptbot

Cooking Tasks

With the goal of developing fully autonomous cooking robots, building robust systems that can chop a wide variety of objects is important. We developed an autonomous framework to sequence together action primitives for the purpose of chopping fruits and vegetables on a cluttered cutting board. We present a novel technique to leverage vision foundational models SAM and YOLO to accurately detect, segment, and track fruits and vegetables as they visually change through the sequences of chops, finetuning YOLO on a novel dataset of whole and chopped fruits and vegetables.

Google Sites

Report abuse