Reviewer Instructions

Review Process

The review process will involve the following stages:

• Artifact submission and bidding: The reviewers will be encouraged to bid on artifacts that they are interested and qualified to evaluate. Once the artifacts are assigned to reviewers, they are expected to read the paper and understand the technical details needed to reproduce the results.

• Kick-the-tires Deadline: Within a couple of weeks, the reviewers will run the basic commands / tests needed to ensure that the artifact compiles and runs, and the documentation is sufficient to do so. This is helpful to identify at an early stage whether the authors have provided all the required code and sufficient instructions, or if the reviewer has all of the software and hardware requirements needed to run the artifact. 

• Q&A With Authors: During the process, the reviewers are encouraged to communicate with the authors about encountered issues immediately (and anonymously) via HotCRP to give the authors time to resolve all problems. Note that our philosophy of artifact evaluation is not to fail problematic artifacts but to help the authors improve their artifacts and pass the evaluation! The authors are also encouraged to push updates as needed to the artifact to enable easy reproduction of the results.

• Final Review Due: At the end, the reviewers will provide a detailed review of the artifact. The focus would be to document the steps followed to run the artifact, and highlight the results they were or were not able to reproduce. The reviewers can also highlight any aspects of the artifact that wow'ed them and helped reproducibility, or any issues they faced in the process. Note that this is not a review of the paper itself, but of the submitted artifact code or data. Finally, the reviewers will assign scores to artifact to decide whether the requested badges (Availability, Functional, and Reproducible) may be awarded. Exceptional artifacts can also be nominated for the Distinguished Artifact honor.

A detailed guide for evaluating artifacts and assigning badges is provided in this Artifact Evaluation Guide by Grigori Fursin, cTuning.

Interested in Reviewing Artifacts?

If you would like to be a part of the Artifact Evaluation Committee for ASPLOS'24, or know someone who would be a good fit, please check out the (self-)nomination page for the AEC at this link by July 11, 2023. (CLOSED)

Tentative Deadlines

Expected Time Commitment

• Reviewers on the Artifact Evaluation Committee will be expected to serve for all three submission cycles.

• In each cycle, we expect one reviewer will review 2-3 artifacts on average, which would require 10-15 hours in total.

• The assigned reviews are expected to be in the area of expertise of the reviewer, so this would rather be quite synergistic with the reviewer's research.