Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

We introduce an algorithmic framework to analyze concurrent environments in continuous and discrete time. When actions are allowed to be executed concurrently in standard Markov Decision Processes, we show that the increased partial observability of the environment causes learning challenges for the policy. However, by providing value-based reinforcement learning methods with additional information about the state delays in the environment, concurrent knowledge models maintain theoretical Q-learning convergence guarantees. We compare concurrent knowledge feature representations that include previous action, inference heartbeat (H), action selection time (t_AS), vector-to-go (VTG), and timestep in the continuous time case (t).

We evaluate our method on toy ablation study MuJoCo tasks, large scale simulated grasping, and real world grasping. Concurrent knowledge models are able to achieve comparable task success to baseline methods while acting 37% faster. As seen in the demonstration videos, concurrent knowledge models are able to act quickly and smoothly without the need to pause and think between actions.