Artifact Appendix
Download Link
Download the ae.tex template for the artifact appendix here. You may also format the artifact appendix yourself, provided it contains all the relevent sections in the ae.tex file.
Artifact Appendix Overview
Next, we describe what each section in the artifact appendix should cover. Please read this carefully to make sure that the artifact appendix provides enough information to reproduce the results in the paper.
Checklist [Obligatory] (click to expand)
Fill in whatever is applicable with some informal keywords and remove all items that are unrelated to the artifacts. Please consider questions below just as informal hints that will help reviewers figure out what to watch out for when evaluating the artifact.
Algorithm: Are you presenting a new algorithm?
Program: Which benchmarks do you use (PARSEC ARM real workloads, NAS, EEMBC, SPLASH, Rodinia, LINPACK, HPCG, MiBench, SPEC, cTuning, etc)? Are they included or should they be downloaded? Which version? Are they public or private? If they are private, is there a public analog to evaluate your artifact? What is the approximate size?
Compilation: Do you require a specific compiler? Public/private? Is it included? Which version?
Transformations: Do you require a program transformation tool (source-to-source, binary-to-binary, compiler pass, etc)? Public/private? Is it included? Which version?
Binary: Are binaries included? OS-specific? Which version?
Model: Do you use specific models (ImageNet, AlexNet, MobileNets)? Are they included? If not, how to download and install? What is their approximate size?
Data set: Do you use specific data sets? Are they included? If not, how to download and install? What is their approximate size?
Run-time environment: Is your artifact OS-specific (Linux, Windows, MacOS, Android, etc) ? Which version? Which are the main software dependencies (JIT, libs, run-time adaptation frameworks, etc); Do you need root access?
Hardware: Do you need specific hardware (supercomputer, architecture simulator, CPU, GPU, neural network accelerator, FPGA) or specific features (hardware counters to measure power consumption, SUDO access to CPU/GPU frequency, etc)? Are they publicly available?
Run-time state: Is your artifact sensitive to run-time state (cold/hot cache, network/cache contentions, etc.)
Execution: Any specific conditions should be met during experiments (sole user, process pinning, profiling, adaptation, etc)? How long will it approximately run?
Metrics: Which metrics are reported (execution time, inference per second, Top1 accuracy, static and dynamic energy consumption, etc) - particularly important for multi-objective benchmarking, optimization and co-design (see ACM ReQuEST ML/SW/HW co-design tournaments).
Output: What is your output (console, file, table, graph) and what is your result (exact output, numerical results, measured characteristics, etc)? Is expected result included?
Experiments: How to prepare experiments and replicate/reproduce results (OS scripts, manual steps by user, IPython/Jupyter notebook, automated workflows, etc)? Do not forget to mention the maximum allowable variation of empirical results!
How much disk space required (approximately)?: This can help evaluators and end-users to find appropriate resources.
How much time is needed to prepare workflow (approximately)?: This can help evaluators and end-users to estimate resources needed to evaluate your artifact.
How much time is needed to complete experiments (approximately)?: This can help evaluators and end-users to estimate resources needed to evaluate your artifact.
Publicly available?: Will your artifact be publicly available? If yes, we may spend an extra effort to help you with the documentation.
Code licenses (if publicly available)?: If your workflows and artifacts will be publicly available, please provide information about licenses. This will help the community to reuse your components.
Data licenses (if publicly available)?: If your workflows and artifacts will be publicly available, please provide information about licenses. This will help the community to reuse your components.
Workflow frameworks used?: Did authors use any workflow framework which can automate and customize experiments?
Archived?: Note that the author-created artifacts relevant to this paper will receive the ACM "artifact available" badge *only if* they have been placed on a publicly accessible archival repository such as Zenodo, FigShare or Dryad. A DOI will be then assigned to their artifacts and must be provided here! Personal web pages, Google Drive, GitHub, GitLab and BitBucket are not accepted for this badge. Authors can provide this link at the end of the evaluation.
How to access [Obligatory]
Describe how reviewers can access your artifact. Here are some common access procedures:
Clone repository from GitHub, GitLab, BitBucket or any similar service.
Download package from a public website.
Download package from a private website. Because the review process is single-blind, you can provide the full link in the appendix.
Access artifact via private machine with pre-installed software. This is only acceptable when access to rare hardware is required or proprietary software is used - you will need to send information and credentials to access your machine to the AE chairs.
Please describe approximate disk space required after unpacking your artifact (to avoid surprises when artifact requires 20GB of free space). We do not have a strict limit but strongly suggest to limit the space to several GB and avoid including unnecessary software packages to your VM images.
Dependencies [Obligatory]
First the artifact appendix should provide some basic information to help the user assess if their system can run the the supplied artifact. This section outlines any hardware and software dependences and any datasets and models that are used in the evaluation.
Hardware dependencies [Optional]: Describe any relevant hardware requirements. This may describe specialized hardware requirements, specialized resource requirements for general purpose hardware, or specialized measurement equipment (e.g., Oscilloscope, Digital Analyzer).
Examples: CUDA-enabled NVidia GPUs, Xilinx Virtex 7 FPGA, Machine with > 32 cores and > 128 GB memory
Hint: If you require specialized, commercially available hardware (e.g., GPUs & FPGAs), it may be helpful to identify several potential devices your system can be evaluated on, as we may have trouble finding a reviewer who has your exact hardware model.
Hint: We do not yet have a funding mechanism for providing AWS credits to reviewers, so we recommend avoiding AWS nodes as a hardware target if at all possible.
Software dependencies [Optional]: Describe any specific OS and software packages required to evaluate your artifact. This is particularly important if you share your source code and it must be compiled or if you rely on some proprietary software that you can not include to your package. In such case, we strongly suggest you to describe how to obtain and to install all third-party software, data sets and models.
Examples: Xilinx Vivado, Cadence Virtuoso, Tensorflow, BLAS
Datasets [Optional]: Describe any datasets that the artifact depends on. If third-party data sets are not included in your packages (for example, they are very large or proprietary), please provide details about how to download and install them. In case of proprietary data sets, we suggest you provide reviewers a public alternative subset for evaluation.
Examples: Machine Learning Datasets, Datacenter Data, SPEC Benchmark Suites, Fabrication Facility Data, Raw Measurement Data (from Oscilloscope, for example)
Models [Optional]: Describe any models that the artifact depends on. If third-party models are not included in your packages (for example, they are very large or proprietary), please provide details about how to download and install them.
Examples: Proprietary process design kits (PDKs), AlexNet, ImageNet
Installation and Testing
Next, the artifact appendix should outline how to install the artifact, and how to navigate the project directories to find any relevant inputs, outputs and source files. The authors conclude this section with a basic test command that tests the basic functionality of the artifact -- this command will be used used to validate that the artifact is set up in the "kick-the-tires" phase of reviewing.
Installation [Obligatory]: Describe how to set up the artifact for evaluation. We recommend you make this process as streamlined as possible (through Docker, for example) and should target novice users.
Basic Test [Obligatory]: Provide a single command that end-users can run to make sure the artifact is set up properly and generating reasonable outputs. Describe what outputs the authors should see if the system worked correctly. For example, a compiler artifact may have a basic test that involves compiling a hello world script and running the compiled output.
Evaluation
Finally, the artifact appendix should describe how to reproduce the results presented in the evaluation section of the paper. This section may also provide instructions for how the user can customize the artifact to run their own experiments.
Experimental Workflow [Recommended]: Describe the experimental workflow and how it is implemented, invoked and customized (if needed). This may involve describing relevant scripts, test harnesses, and interactive evaluation environments (e.g., Jupyter/IPython notebooks, portable CK workflow) used in the evaluation. If applicable, overview any relevant output directories, source directories, and point out where to find pertinent log files and temporary files to aid in debugging. If done well, an end-user should be able to deeply investigate how the artifact operates and how the analysis is performed.
Evaluation and Expected Results [Obligatory]: Walk through how to reproduce the results in the paper with the packaged artifact. Effective evaluation sections link output data files to tables, graphs, and empirical values in the paper. Describe the expected result and the maximum allowable variation of empirical results (particularly important for performance numbers and speed-ups). See the SIGPLAN Empirical Evaluation Guidelines and the AE FAQ.
Experiment customization [Optional]: Describe how the end-user could modify the artifact to run their own experiments. For example, if the artifact is a compiler, this section may describe how to write and compile their own program. It is currently optional since it is not always trivial. If possible, describe how to customize your workflow, i.e. if it is possible to use different data sets, benchmarks, real applications, predictive models, software environment (compilers, libraries, run-time systems), hardware, etc. Also, describe if it is possible to parameterize your workflow (whatever is applicable such as changing number of threads, applying different optimizations, CPU/GPU frequency, autotuning scenario, model topology, etc).