PyHyFlex Framework (HyFlex 3.0)

Version: v1.4

Date: 2024-07-29

Authors: Chaofan Tu / Weiyao Meng

Background

Meta-heuristics are well-known for solving a variety of complex optimisation problems, ranging from bin packing, nurse rostering, and timetabling to vehicle routing problems [1]. Researchers are constantly seeking more efficient and effective methods for solving cross-domain optimisation problems, leading to more research into the automated design of meta-heuristics [2]. This surge in research has led to the development of different platforms and frameworks aimed at facilitating the automated algorithm design process.

One of the widely used tools in the automated design of meta-heuristics is HyFlex [3, 4], a Java-based cross-domain computing framework notable for its hyper-heuristic algorithm structure [5]. The HyFlex framework, as illustrated in Figure 1, segregates the problem domain from the algorithm space, reducing dependency on specific problem details and allowing researchers to focus on algorithm design. The current HyFlex framework supports six classic combinatorial optimization problems, with extensions for an additional two problems [6]. It has been considered for the Cross-domain Heuristic Search Challenge (CHESC).

Automated algorithm design has gathered increasing attention from researchers across various domains, leading to the application of different methods, particularly in the domain of machine learning. The field of research in automated design of meta-heuristics has received substantial development [7], from traditional machine learning techniques to advanced deep learning methods, and even large language models. These methods, applied in optimisation, have demonstrated a data-driven paradigm for solving combinatorial optimization problems, known as 'Learn to Optimize' [8, 9]. This paradigm leverages the power of data to improve the efficiency and effectiveness of optimization processes, making it a crucial area of study in modern computational research.

Figure 1. the HyFlex framework [3].

The flexibility and extensibility of HyFlex make it a popular choice in the literature for benchmarking and developing new optimization techniques [4, 10]. However, despite HyFlex's strength in cross-domain search, most modern machine learning libraries and tools are predominantly developed in Python. HyFlex's reliance on Java increases the complexity and implementation costs of interaction with Python ecosystems. This language disparity creates a significant barrier for researchers and practitioners who wish to leverage Python's rich machine learning libraries within the HyFlex framework.

Introduction

Recognizing the challenge of integrating HyFlex with modern machine learning tools, the PyHyFlex project aims to bridge the gap between HyFlex and Python, thereby enhancing research and development in cross-domain automated algorithm design.

The objective of PyHyFlex is to develop a user-friendly Python interface that retains HyFlex's original functionality and flexibility while enabling the seamless integration of Python-based machine learning methods within the HyFlex framework.

By simplifying the implementation of Python in HyFlex, various Python machine learning libraries can be integrated within the HyFlex framework, facilitating more efficient and effective automation methods. This integration will provide researchers and practitioners with a powerful tool to leverage state-of-the-art machine learning techniques in their automated algorithm design process. It also paves the way for more advanced data-driven approaches to cross-domain optimization, ultimately contributing to the advancement of optimization research.

Core Features

Seamless Execution in Jupyter Notebook

Using Python to call Java for executing binary code typically requires Py4j, which involves starting a message listening port for each Java execution process, adding complexity. PyHyFlex, however, employs Pyjnius to call the HyFlex Java runtime environment. Unlike Py4j, Pyjnius does not require a background service gateway, making it more efficient.

Additionally, this project modifies Pyjnius to be compatible with Jupyter Notebook, enhancing usability. This modification overcomes the original limitation of Pyjnius, which does not support execution in Jupyter Notebook, thus broadening its applicability.

Gym Environment Interface Compatibility

The project implements the HyFlex instance run process as a Gym environment: the HyflexEnv class, including interfaces such as reset(), step(), render(), and close(), leveraging a wide range of reinforcement learning algorithms designed for Gym environments.

Compared to the Gym class maintained by OpenAI, the HyFlex environment interface is adapted to the Gymnasium class (an upgraded version of Gym), with modifications such as increasing the step() return variables to five. This allows using the Monitor component from the SB3 community to monitor the training process of deep reinforcement learning agents.

Parallel Execution Acceleration (Vectorized Env)

Directly calling the HyflexEnv class can only utilize a single CPU core, making the training process extremely time-consuming. To improve efficiency, this project achieves parallel accelerated execution on multiple CPU cores by wrapping the HyflexEnv environment class with multiple auxiliary methods.

Figure 2 shows the parallel performance evaluation with the execution time for 10,000 steps on a knapsack problem instance using a 64-core CPU server. For one Knapsack instance, the Deep Q-Network (DQN) training process, as shown in Figure 3(b), encompassed a total of 2 million steps. This training was accomplished in 166.67 minutes using 12 CPU cores.

Figure 2. Parallel Performance Evaluation

(a)

(b)

(c)

Figure 3. Reward Trend in DQN Training for one Knapsack Instance.

Panels (a) and (b) differ in the total number of training episodes, with each episode consistently comprising 1000 steps.

Panel (c) displays the same number of training episodes as panel (b), but each episode was set with 2000 steps.

Bug Fixes and Future Work

Bug Fixes

When calling the HyFlex selection low-level heuristic (LLH), a Null Pointer error occurs, causing the Java process to crash. This issue is due to problems in passing integer variables between Python and Java. The project resolves this by improving the variable passing method in the applyHeuristic() call in Java.

To-Dos

Provide a complete training example, including convergence graphs of the training process
Offer benchmark performance comparisons of different DRL algorithms on various problems

Logs & progress remarks

July02

The use of DQN to the knapsack problem, with appropriate parameter settings such as episode step length, has resulted in converged nerual networks that can surpass random strategy.

May21

Parallel accelerated execution is sucesssfully demonstrated on a 64-core CPU server.

April14

The mini hyflex can be called with python in jupyter notebook by fixing the source codes of pyjinus.

April08

the excuation evironment 'pyhyflex' is ready with all requirements satisfied.

Page updated

Google Sites

Report abuse