Naoto Yoshida [1], Yasuo Kuniyoshi [2,3]
[1] Kyoto University, Kyoto, Japan
[2] The University of Tokyo, Tokyo, Japan
[3] Next Generation Artificial Intelligence Research Center, Tokyo, Japan
The ability of animals to continuously acquire skills in their environment is still a mysterious and amazing form of intelligence. All skills necessary for survival are integrated into a single agent, and these are constructed solely from the agent’s internal motivation. In this way, is it possible to enable continual acquisition of behaviors in autonomous agents that learn behavior based solely on bodily information and motivation? This research provides a positive answer to this question by demonstrating the surprising ability to acquire behavior that is inherent in homeostatic reinforcement learning (homeostatic RL), a bio-inspired method proposed in computational neuroscience. Homeostatic RL focuses on the rich dynamics resulting from the coupling of the environment and the internal dynamics of the body, which are usually ignored in regular RL. This allows the agent to exhibit a variety of emergent abilities, despite the seemingly simple error minimization problem for the internal state of the body (homeostasis).
In this study, we apply homeostatic RL to the Crafter, an open-ended environment in deep RL. We implemented homeostatic RL through motivation based on homeostasis for Health, Hunger, and Thirst, and Energy (Figure A). We constructed a policy consisting of a convolutional neural network (Resnet) and RNN, and used images and interoception as inputs (Figure B). We used Asynchronous PPO to optimize the training for 5B steps, and obtained an agent that could survive for up to 10,000 steps (right panel, red lines), the maximum number of steps for Crafter. We oberved that the agent could acquire various survival skills such as foraging and water collection, as well as attacking enemies and building shelters, in an integrated manner with the aim of maintaining homeostasis within the body.
Behaviors of Homeostatic Agent and Original Crafter-reward Agent
Homeostatic Agent Trained in Crater for 5B Steps
Original Crafter Reward Agent Trained in Crater for 5B steps
Foraging Behaviors Directly Associated with Homeostasis
Water Collection
Hunting Cow (Food Collection)
Complex Behaviors Emerged through Optimization for Homeostasis
Beating Zombie
Skillful Escape from Skeletons
Building Shelter
Building Bridge-like Objects
Emergence of diverse behaviors observed repeatedly
Code available from: https://github.com/ugo-nama-kun/openended_homeostatic_rl
Reference
Yoshida, N. & Kuniyoshi, Y. (2025) Unexpected Capability of Homeostasis for Open-ended Learning, 2025 IEEE International Conference on Development and Learning (ICDL) (accepted)
