MLArchSys 2024 Paper Checklist Guidelines
We adopted the guidelines from NeurIPS 2023 and ICLR 2023.
The MLArchSys Paper Checklist is designed to encourage best practices for responsible research, addressing issues of reproducibility, transparency, research ethics, and societal impact. For all authors:
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Claims in the paper should match theoretical and experimental results in terms of how much the results can be expected to generalize.
The contributions should be clearly stated in the abstract and introduction, along with any important assumptions and limitations. It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
(b) Did you discuss any potential negative societal impacts of your work?
If you see a direct path to any negative applications, you should point it out, even if it's not specific to your work. Consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, you could also discuss any mitigation strategies.
(c) Did you describe the limitations of your work?
You are encouraged to create a separate "Limitations" section in your paper.
Point out any strong assumptions and how robust your results are to violations of these assumptions. Reflect on how these assumptions might be violated in practice and what the implications would be.
Reflect on the scope of your claims, e.g. if you only tested your approach on a few datasets or did a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated in details.
Reflect on the factors that influence the performance of your approach. For example, when
We understand that authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejections. It is worth keeping in mind that a worse outcome might be if reviewers discover limitations that aren't acknowledged in the paper. In general, we advise authors to use their best judgement and recognize that individual actions in factor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
(d) If you ran experiments, did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? You may check the NeurIPS code and data submission guidelines for more details. While we encourage release of code and data, we understand that this might not be possible, so no is an acceptable answer. Papers can not be rejected simply for not including code, unless this is central to the contribution (e.g. for dataset track or a new open-source benchmark). At submission time, to preserve anonymity, remember to release anonymized versions.
(e) If the contribution is a dataset or model, what steps did you take to make your results reproducible or verifiable? Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a new accelerator, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), release of a model checkpoint, or other means that are appropriate to your research.