Lab 14

Lab 14: Human-in-the-loop Robot Learning

In this lab you will formulate (and optionally implement) the problem of teaching your Cozmo robot how to clean up a table by pushing objects out of the way as a human-in-the-loop learning problem.

1. Previous labs involved clearly specified planning or reinforcement learning problems where you worked to implement a solution. However, often times in robotics the key challenge is to express a practical robotics problem within a well-studied formalism such as reinforcement learning. This involves choosing the right framework (planning or RL? model-free or model-based? human input?) as well as defining states and actions, modeling transitions and rewards, among other things.

For this lab you will formulate the problem of pushing objects on a table into a desired location (dust bin) as a human in the loop RL problem. First answer the following questions:

    • What are the robot's states and actions? You can think of the table as a grid, but you do not have to.
    • How would you specify the goal state in the robot's state space?
    • What are the expected sources of uncertainty?

Then choose one of the human-in-the-loop learning frameworks covered in class for solving the problem of teaching the robot how to clear the table. Justify your choice and specify all components of the framework you chose, such as: other components of the MDP, (if demonstrations are involved) what information is captured from demonstrations, (if human rewards are involved) what is the interface for providing rewards, (for any modeling problem) what are the models and learning algorithms used, etcetera.

Note that unlike most previous labs this is an open-ended design question and there are no right or wrong answers. However, your answer should be complete as to allow someone to implement your solution and your reasoning for design choices should be sound.

2. (Optional) Given the strict timeline for this last lab there is no required implementation. However, you probably have almost all pieces of your proposed solution already implemented at this point. Hence we encourage students to implement their solution formulated in the first question on the Cozmo robot. You can use the robot's cubes as the objects to be removed from the table.