Hai Nguyen*, Long Dinh^, Robert Platt*, Christopher Amato*
* Khoury College of Computer Sciences, Northeastern University, Boston, MA 02115, USA
^ Horus AI, Hanoi, Vietnam
Email: nguyen.hai1@northeastern.edu
Even though partial observability is prevalent in robotics, most reinforcement learning studies avoid it due to the difficulty of learning a policy that can efficiently memorize past events and seek information. Fortunately, in many cases, learning can be done in an asymmetric setting where states are available during training but not during execution. Prior studies often leverage the state to indirectly influence the training of a history-based actor (actor-critic methods) or a history-based critic (value-based methods). Instead, we propose using state-observation and state-history mutual information to improve the agent's architecture and ability to seek information and memorize efficiently through intrinsic rewards and an auxiliary task. Our method outperforms strong baselines through extensive experiments and achieves successful sim-to-real transfers to a real robot.
Previous approaches use states as extra information, e.g., using the state as an additional input to learn a value function, thus better learning a history-based policy.
Instead, we directly use the mutual information between history (containing the knowledge so far) and the state as a source of intrinsic rewards to encourage information gathering. The intuition is that we want to encourage actions that lead to more knowledge about the state (i.e., to maximize I(h; s)).
We use a two-staged approach (see below)
Given a dataset with states (e.g., generated from a random agent or demonstrations), we want to learn state (z^s) and observation (z^o) features that are:
Task-relevant: Irrelevant features should be filtered out. We use the bisimulation framework for this purpose, i.e., use z^s and z^o to predict the reward and the next state and observation features.
Non-overlapping: we want to minimize I(z^s; z^o) to reduce information overlap, as later, we will use z^s + z^o for task learning.
No prevent trivial solution, we also maximize I(z^s; s) and I(z^o; o).
We proposed a two-architecture branch with a memory-based top branch and a memoryless bottom branch.
The top branch is supposed to output state features not trivially observed from the observation (i.e., z^s). Therefore, we use z^s to supervise the output of the top branch.
We use z^o to initialize the weight of the bottom branch.
We use I(h;z^s) to compute intrinsic rewards to encourage information gathering.
