Physics Engines in Robotics

In this post, I will discuss my recent thoughts on physics engines and simulators for robotics, with a focus on motion planning and deep learning. Disclaimer: I know very little about how physics engines are designed or written but I have many of them so take my poorly-informed thoughts with a grain of salt.

Physics engines and simulators are a big deal in robotics because they let us test things without the risk or time cost of a real robot, and they have several major advantages. However, for learning I would say there are four challenges which I find interesting.

The first issue I want to highlight is the speed of physics engines. The main physics engines, like Bullet, ODE, PhysX, MuJoCo are all highly optimized and are impressively fast. In fact, they can simulate a full humanoid in real time on a nice desktop computer. However, that's doesn't mean they are fast enough to use for real-time motion planning. In robotic manipulation especially, we would need a physics simulator to be 10 or 100 times real-time speed in order to use it inside of a real-time planning loop. This is because the algorithms that we use in robotic manipulation like RRT and PRM will make on the order of 10,000 to 100,000 calls to our physics engine. If we want to replan once per second (a reasonable ask in my opinion) we would need a physics engine that can simulate 10-100,000 steps in one second.

Now, that's a pretty naive estimate, and there is existing work which uses physics engines inside a planning loop [1]. But overall I think it's fair to say that we still have a long way to go before we could use a simulator inside of a planner for complex scenarios.

My second point is that traditional physics engines are not differentiable. This means, for example, that you cannot compute how to change the forces applied to change the position of an object. Having that information would be incredibly helpful for training agents, because it would mean you could use gradient descent to arrive at controls that move you in a specific direction (useful for motion planning algorithms) or you could use it to better estimate how your reward changes as a function of actions (useful for reinforcement learning). This is already a hot topic in the RL community, so called "Model Based RL", and within the control communities these problems have been studied for decades. Still, again it is reasonable to say that none of the major general-purpose physics engines suited for robotics offer any such capabilities.

This is perhaps the problem for which I have the least ideas on solving. How do we get a model of the world into a simulator in the first place? It's fine for a human to do this part if you're dealing with a task which changes infrequently (manufacturing) but for most interesting tasks this is incredibly time-consuming and requires lots of trial and error on the part of the human.

My last point is that there still does not exist a physics engine that is good at simulating all phenomena. The new FleX engine by NVIDIA looks very promising, but still there are things it cannot simulate (the torsion of a rope for instance). Therefore, when we train in one of these simulators, we must be aware that there are some physical phenomena about which our agent will not be able to learn. We will have the same problem if we are training a neural network to based on simulated data.

Having described four of the issues with the state of physics simulators for robotics, I'd like to highlight some recent work which I think is working to address these issues.

First, there have been a number of papers in the last few years on writing physics engines using auto-differentiation packages like Theano [2] or PyTorch [3]. With this approach, we get gradients or approximate gradients that we can reason about just like any other mathematics -- there is no learning involved. However, there is another approach where you simply design a neural network structure that you think nicely represents the physics you want to learn, and then you train on the data from an existing simulator [4, 5, 6, 7]. This is another interesting approach, because it has the potential to be trained on data from a variety of sources, including (most importantly) real world interaction!

Citations & Resources

[1] Muhayyuddin, Moll, M., Kavraki, L., & Rosell, J. (2017). Randomized Physics-based Motion Planning for Grasping in Cluttered and Uncertain Environments, (1). https://doi.org/10.1109/LRA.2017.2783445

[2] Degrave, J., Hermans, M., Dambre, J., & Wyffels, F. (2016). A Differentiable Physics Engine for Deep Learning in Robotics. Frontiers in Neurorobotics, 13, 1–8. https://doi.org/arXiv:1611.01652v2

[3] de Avila Belbute-Peres, F., Smith, K., Allen, K., Tenenbaum, J., & Kolter, J. Z. (2018). End-to-end differentiable physics for learning and control. Advances in Neural Information Processing Systems, (NeurIPS 2018), 7178–7189. Retrieved from https://papers.nips.cc/paper/7948-end-to-end-differentiable-physics-for-learning-and-control

[4] Author, A. (n.d.). Interaction Networks for Learning about Objects, Relations and Physics. Retrieved from https://papers.nips.cc/paper/6418-interaction-networks-for-learning-about-objects-relations-and-physics.pdf

[5] Byravan, A., & Fox, D. (2016). SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks. Retrieved from http://arxiv.org/abs/1606.02378

[6] Chang, M. B., Ullman, T., Torralba, A., & Tenenbaum, J. B. (2016). A Compositional Object-Based Approach to Learning Physical Dynamics. Retrieved from http://arxiv.org/abs/1612.00341

[7] Mrowca, D., Zhuang, C., Wang, E., Haber, N., Fei-Fei, L., Tenenbaum, J. B., & Yamins, D. L. K. (2018). Flexible Neural Representation for Physics Prediction. https://doi.org/arXiv:1806.08047v1