Hokoff

The advancement of Offline Reinforcement Learning (RL) and Offline Multi-Agent Reinforcement Learning (MARL) critically depends on the availability of high-quality, pre-collected offline datasets that represent real-world complexities and practical applications. However, existing datasets often fall short in their simplicity and lack of realism. To address this gap, we propose \texttt{Hokoff}, a comprehensive set of pre-collected datasets that covers both offline RL and offline MARL, accompanied by a robust framework, to facilitate further research. This data is derived from Honor of Kings, a recognized MOBA game known for its intricate nature, closely resembling real-life situations. Utilizing this framework, we benchmark a variety of offline RL and offline MARL algorithms. We also introduce a novel baseline algorithm tailored for the inherent hierarchical action space of the game. We reveal the incompetency of current offline RL approaches in handling task complexity, generalization and multi-task learning.

• The tasks we adopt are based on one of the world's most famous Multiplayer Online Battle Arena (MOBA) games, Honor of Kings (HoK), which has over 100 million daily active players, ensuring the practicality of our datasets. The complexity of this environment dramatically surpasses those of its counterparts, demonstrating the potential for simulating real-world scenarios more accurately.

• We present an open-source, easy-to-use framework. This framework includes comprehensive processes for offline RL (sampling, training, and evaluation), and some useful tools. Based on the framework, We release a rich and diverse set of datasets with different design factors, catering not only to offline RL but also offline MARL.

• Building on the framework, we reproduce various offline RL and offline MARL algorithms and propose a novel baseline algorithm tailored for the inherent hierarchical structured action space of Honor of Kings. We fully validate and compare these baselines on our datasets. The results indicate that current offline RL and offline MARL approaches are unable to effectively address complex tasks with discrete action space. Additionally, these methods exhibit shortcomings in terms of their generalization capabilities and their ability to facilitate multi-task learning.