The optimization decisions that compilers make are critical for the performance and efficiency of compiled software. However, tuning and optimizing compilers is hard. There are many decisions to make, a near infinite number of possible programs, and most programs respond differently to optimizations. Performing this task by hand is time consuming, error prone, and yet often leads to suboptimal results. As a result, interest in applying AI techniques to this task is rapidly increasing.
The goal of this competition is to develop an agent for a classic problem in compilation, phase ordering to minimize code size. The phase ordering problem is to select, from a candidate set of optimization passes, which to run on an unseen input so as to produce the best output. Simple black box optimizations achieve strong results in minutes to hours of compute. The challenge is to achieve similar or smaller code size in less time. We provide training sets, starter code, and scripts to evaluate and validate results. The task is well-defined and suitable for a range of AI approaches. There is a broad scope of research opportunities, from gradient-free optimization to deep reinforcement learning.
We are using CompilerGym v0.2.3 to run the competition. The goal is to develop an agent for the "llvm-ic-v0" environment that minimizes code size by maximizing the cumulative reward on an unseen program. You may use any datasets (including your own) for developing and training your agent, and you may use any observation spaces (including your own). See "Useful Links" below for some pointers and examples.
Your agent will be evaluated on a set of unseen LLVM bitcodes that will test the generalization of your approach. To encourage both submissions that use machine learning models and conventional heuristic techniques, we have two categories for evaluation:
"Short": This allows a maximum of 5 resets and 1000 steps in total for each evaluation of an unseen program.
"Long": This allows a maximum of 1k resets and 1m steps in total for each evaluation of an unseen program.
There are no constraints on hardware or compute resources.
1. Download this evaluation script. Verify the integrity of the downloaded script using:
$ sha256sum mlarchsys2022.py
53faa680830cf318fcedca8303360cc78cf72d1ee4d2acb9a11db04dc6bce588 mlarchsys2022.py
Note: Our evaluation script supports only Linux, not macOS. If you require macOS support please leave a comment here.
2. Open the script in the editor of your choice and read the documentation at the top. This tells you how to modify the script to plug in your agent, how to configure the script to run either the "short" or "long" evaluation category, and other configuration options that affect how the evaluation is run.
3. Run the script by following the instructions in the comments. We remind you that once you have run the evaluation script and generated a set of test results, no further modifications are allowed to your agent.
4. Submit your results to our OpenReview. Please include the following:
The CSV results file that is generated by the evaluation script in step (3).
A write-up of your approach (up to 4 pages), describing the algorithm used, any parameters tuned, etc. Please follow the ISCA'22 Latex Template.
We strongly encourage you to include a link to a GitHub repository with the implementation of your agent, though this is not required.
The deadline for submissions is May 31st June 5th, Midnight Anywhere on Earth. We will review the submissions and, if necessary, follow up with clarifying requests.
The winner in each of the two categories will be announced in the workshop on June 19th, and the full table of results and write-ups will be available online after the event.
For technical support, please file a bug report on GitHub and put "[MLArchSys2022]" at the start of the subject line. For any other questions, please get in touch. Good luck!
We have prepared a short demo notebook.
The Getting Started guide introduces the concepts and terminology.
The CompilerGym LLVM Instruction Count leaderboard is similar to the evaluation script. that we will be using. You may find this useful for developing your agent.
The examples directory of the CompilerGym repository includes several reference implementations of reinforcement learning and auto-tuning approaches that can be used as a starting point for developing your agent.
Competition Starts: March 21, 2022
Test Set Release: May 17th, 2022
Submission deadline: May 31st June 5th, 2022 - Midnight Anywhere on Earth
Workshop: June 19th, 2022
Please join Compiler Optimization MLArchSys 2022 Google Group to chat with the organizers and other competitors. For technical support with CompilerGym, please use the issue tracker.