2013.06.19: Marc Deisenroth - Bayesian Machine Learning for Controlling Autonomous Systems

Deisenroth & Rasmussen (ICML, 2011): PILCO

Deisenroth et al. (PAMI): Gaussian processes for data-efficient learning in robotics and control

How do you control systems for which you do not have a good model for?

• Bayesian Machine Learning

We want to learn policies fully autonomously:

• Infinite number of combinations of state and control
• Millions of experiments are impractical

We could use three approaches to learn a control

• Reinforcement learning
• Imitation learning
  • Inverse RL - Ng & Russell 2000
  • Behavioral cloning - Pomerleau 1989
  • Probabilistic imitation learning - Interesting that the "guidance" can actually affect the learning.
• Bayesian optimization
  • Jones 2011
  • Brochu et al 2010
  • Hennig & Schuler 2012
  • Similar to active learning
  • Calandra - Seyfarth : Bipedal robot
  • Limited to 10 - 20 parameters
    1. Build a model of the objective function
    2. Find the minimum
    3. Evaluate the true objective function
    4. Update the model objective function

Reinforcement learning

• x_t+1 = f(x_t, u_t) + w
• u_t = p(x_t, z)
• min J(z)
  1. Probabilistic model for transition function "f" to be robust to model errors
     1. Gaussian process: mean and covariance function
  2. Compute long-term predictions of p(x,z)
  3. Policy improvement
  4. Apply controller