EMOGI

Generating Behavior-Diverse Game AIs with Evolutionary Multi-Objective Deep Reinforcement Learning

Abstract

Designing artificial intelligence for games (Game AI) has been long recognized as a notoriously challenging task in game industry, as it mainly relies on manual design, requiring plenty of domain knowledge. More frustratingly, even spending a lot of efforts, a satisfying Game AI is still hard to achieve by manual design due to the almost infinite search space. The recent success of deep reinforcement learning (DRL) sheds light on advancing automated game designing, significantly relaxing human competitive intelligent support. However, existing DRL algorithms mostly focus on training a Game AI to win the game rather that the way it wins (style). To bridge the gap, we introduce Evolutionary Multi-Objective Game Intelligence (EMOGI), an end-to-end game design framework, leveraging the power of evolutionary multi-objective optimization (EMO) and DRL to generate behavior-diverse Game AIs with barely prior human knowledge. Firstly, EMOGI proposes the style-oriented learning to bypass manual reward shaping in DRL and directly learns a Game AI with an expected style in an end-to-end fashion. On this basis, the prioritized multi-objective optimization is introduced to achieve more diverse, nature and human-like Game AI. Large-scale evaluations on a Atari game and a commercial massively multiplayer online game are conducted. The results demonstrates that EMOGI, compared to existing algorithms, achieve better game designs in an intelligent and automatic way.

Framework

Fig. 1: The Overall framework of EMOGI

From a higher perspective, EMOGI is built on an evolutionary process consists of two parts: Asynchronized Evolution and Synchronized Selection (As shown in Fig. 1). In the beginning, EMOGI initializes and maintains a population of candidates (black dots). Then, in asynchronized evolution, all candidates are mutated and evolved to create off- spring (new styles) through automatic reward shaping, and the newborns’ style will be evaluated by environment interaction (game playing). After evaluation, in the synchronized selection, only better offspring will be kept for the subsequent evolution while the rest be eliminated.