Artifact Evaluation
About the Artifact Evaluation Process
This year, AIIDE will include an Artifact Evaluation, a chance for authors of accepted papers & posters to submit a companion software artifact with their paper. Example types of artifact include stand-alone software, web applications, datasets, plug-ins/extensions for existing tools, social media bots, and others.
The purposes of the artifact evaluation are:
- To promote reproducibility of our research results by reviewing the claims made in the paper and how well they are supported by the corresponding software;
- To promote reuse by encouraging authors to release software that is well-documented and easy to use by peers;
- To recognize software artifacts as important scholarly contributions in addition to the paper itself.
All submitted artifacts will be stringently peer reviewed through a separate review process from the main conference. Artifact evaluation is an optional round of additional reviewing available to authors of accepted papers and posters; the quality of artifacts will not affect acceptance decisions for main conference papers in any way. One companion artifact may be submitted per paper.
FAQ: But don't we already have demos and Playable Experiences? Why introduce another thing?
The Artifact Evaluation is distinct from the Playable Experiences and Demos tracks in that the software artifacts are tightly coupled with a research result documented primarily in a conference paper. Passing the artifact evaluation process is a signal to other researchers in the field that your software is robust, reusable, and that your research claims are reflected in the artifact (thus promoting reproducibility).
To help authors distinguish between these submission types, we summarize their criteria below:
FAQ: Can I submit to artifact evaluation if I haven't submitted to the main conference?
No. Artifact evaluation is only conducted on artifacts associated with accepted papers in the main conference. Authors of accepted papers will be contacted regarding how to submit their artifact for evaluation after paper notifications are sent out.
FAQ: Do all accepted papers need to submit an associated artifact for evaluation?
No. Artifact evaluation is optional, though encouraged, for authors of accepted papers. Artifact evaluation follows a separate peer review process from papers, with a separate (though potentially overlapping) committee of reviewers from the main conference program committee. Artifacts that pass this review process will be publicized on the AIIDE website and announced at the main conference.
FAQ: Where can I learn more about artifact evaluation?
Artifact evaluation is a new process being adopted at conferences across computer science, and it is new for AIIDE in 2018. You can learn more about artifact evaluation here.
Feedback on this process is welcome, and questions can be directed to the artifact evaluation co-chairs or the program chair. More details regarding this process will be made available closer to artifact evaluation submission deadline.