Grant Wang, Tae Kyun Kim, Irving Fang
UC Berkeley EECS C106A: Introduction to Robotics
Welcome to our site that highlights our work on using the Rethink Robotics Baxter robot and vision sensors to identify and collect different types of trash using machine learning and specialized object pickup tools. Below you will find the (1) Introduction, (2) Design, (3) Implementation, (4) Results, (5) Conclusion, (6) Team, and (7) Additional Information.
Trash identification and grasping is a hard problem. At times, trash can be in the form of easily discernable, discrete objects (i.e. cans, bottles), but other times it can be a loose collection of granular objects as well (i.e. bread crumbs, clusters of dust) or objects with easy slippage (i.e. water bottles, cans).
Picking up these objects is another daunting challenge, as different objects are best collected by equally different techniques. Even humans, with highly manipulable hands, often require tools to better pick up trash (i.e. using broom and dustpan to collect crumbs).
To have robots to perform these kinds of tasks with grasping abilities far more limited than humans, significant effort must be put into end-effector design, precise robot manipulations, and accurate object grasping point calculations, all of which require expensive hardware (high-precision robots and accompanying sensors) and complex software (point-cloud mapping of objects, finding appropriate center of mass, determining multiple grasp points for irregular objects).
However, if we have robots use tools to pick up trash, then significantly less resources need to be expended on both hardware and software, as tools will do the work that robots find difficult. We can also generalize the task better to pickup a wider variety of objects, as the robot's manipulation is not limited by its end-effector. The goal of the project is thus the following:
Identify difficult-to-grasp trash, and command Baxter to manipulate tools to pick up the objects.
Real-world application for this project would be autonomous robot cleaning, from outdoor street sweeping and litter cleanup to restaurant table tidying and setup.
The idea of utilizing machine learning to simply identify target objects, and having a general-purpose robot utilize specialized tools to perform the optimal task based on object identification and a simple set of parameters (i.e. general location, not precise 3D mapping, of the object) is ubiquitous and revolutionary, as demonstrated by this project.
Our design criteria was for the Baxter robot to use to identify at least two distinct types of trash, and use at least two types of specialized tools to collect the trash.
Our modular framework, as our system diagram displays, is divided into three distinct systems:
This modular design allowed us to exceed initial design criteria and have the system identify not only four different types of trash, but also the three types of tools as well, with ability to identify significantly more objects by feeding appropriate number of image samples of object type to the training model. In addition, we were able to have Baxter use two distinct tools, with code modularity designed to allow for easy addition of other types of tools (i.e. electromagnets for magnetic objects). And specialized stand-alone tools allowing for easy manipulation by the limited Baxter arms eliminated the need for complex, non-generalizable trash pickup policies.
This proper system design allowed us to abstract away the three distinct subsystems during development, and allowed better team work allocation by having each team member focus on one subsystem, and work together on pre-defining subsystem handles and endpoints for seamless subsystem integration.
In addition, when robustness and reliability of certain tasks were desired, only relevant subsystems had to be improved, without a total overhaul of the entire project. For instance, trash sweeping reliability improved by only working on the structural integrity of the free-standing broom. In another instance when we wanted more trash to be identifiable, we simply had to train the model more with images and make minor adjustments to the policy selection and IK motion planning.
The trade-off, though, to designing such a robust system resulted in us not having as much time as we would have liked to train a larger dataset of trash objects and test a wider range of tools. Nevertheless, we were able to successfully train the model to identify chips, bottle, cans, and the tools, and test manipulation of a broom, dustpan, and sticky tool on the respective trash objects.
As described in the above system diagram, trash pickup goes through the following steps:
More detailed information about the implementation are described below.
We utilized the state-of-the-art Google Object Detection Network to train a model to identify and calculate a two-dimensional bounding box around objects of interest, as shown in the left image.
Conclusion:
Improvements:
UC Berkeley CS student with research in ai/robotics at AUTOLab
Contributions: Computer vision (Kinect rgbd image manipulation + TF), Tools High-level class and Policy selection, integration of subsystems/pipeline
UC Berkeley EECS student and independent game developer
Contributions: IK motion planning, Baxter Low-level arm manipulation class, tools design/fabrication
UC Berkeley DS/Pure Math student with research in NLP at Berkeley Social Interaction Lab and work at Snipfeed
Contributions: Object detection/recognition system and learning model training and testing
Github link: https://github.com/GrantMeAWish/trash_cleaner_baxter
Tensorflow Object Detection API: https://github.com/tensorflow/models/tree/master/research/object_detection
Faster R-CNN model used in the project: http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_v2_coco_2018_01_28.tar.gz
