Papers from the Machine Learning community are supposed to be a valuable asset. They can help to inform and inspire future research. They can be a useful educational tool for students. They are the driving force of innovation and differentiation in the industry, so quick and accurate implementation is really critical. On the research side they can help us answer the most fundamental questions about our existence - what does it mean to learn and what does it mean to be human? Reproducibility, while not always possible in science (consider the study of a transient astrological phenomenon like a passing comet), is a powerful criteria for improving the quality of research. A result which is reproducible is more likely to be robust and meaningful and rules out many types of experimenter error (either fraud or accidental). There are many interesting open questions about how reproducibility issues intersect with the Machine Learning community:
Our aim in the following workshop is to raise the profile of these questions in the community and to search for their answers. In doing so we aim for papers focusing on the following topics:
We will accept both short paper (4 pages) and long paper (8 pages) submissions (not including references). Submissions should be in the NIPS 2018 format. A few papers may be selected as oral presentations, and the other accepted papers will be presented in a poster session. There will be no proceedings for this workshop, however, upon the author’s request, accepted contributions will be made available in the workshop website. Submission are single-blind, peer-reviewed on OpenReview (https://openreview.net/group?id=ICLR.cc/2019/Workshop/RML), and open to already published work.
Workshop Paper Submission Deadline: 5 March 2019 11:59pm EST
Workshop Paper Decision: TBD
Camera Ready Deadline: TBD
Workshop Date: May 6, 2019.
May 6, 2019 ICLR, New Orleans
9:50-9:55, Opening Remarks
9:55-10:30 Invited Talk: Joel Grus, Title: Reproducibility as a Vehicle for Engineering Best Practices
10:30-11:00 Coffee / Poster Session.
11:00-11:30 Invited Talk: Pablo Samuel Castro, Title: Dopamine: thoughts on reproducibility
11:30-12:00 Contributed Talk: Francesco Locatello, Title: Challenging common assumptions in the unsupervised learning of disentangled representations
12:00-12:30 Contributed Talk: Luca Foschini, Title: Reproducibility in Machine Learning for Health
12:30-13:00 Contributed Talk: Brian Lee, Title: Minigo: A Case Study in Reproducing Reinforcement Learning Research
13:00-15:20 Lunch and Conference Keynote
15:20-15:40 Contributed Talk: Arnout Devos, Title: Reproducing Meta-learning with differentiable closed-form solvers
15:40-16:00 Invited Talk: Avital Oliver, Title: How can reproducibility support understanding?
16:00-16:15 Coffee / Poster Session.
16:15-16:30 Invited Talk: Soumith Chintala, Title: A checklist for open-sourcing your code for reproducibility and extensibility
16:30-17:00 Invited Talk: Jessica Zosa Forde, Title: Promoting Science in Machine Learning Research
17:00-17:30 Invited Talk: Dhruv Madeka, Title: On reproducible research for large scale production deployments
17:30-18:30 Panel Discussion with Hugo Larochelle, Nando De Freitas, Joel Grus, Jessica Zosa Forde, Pablo Samuel Castro
More to be added soon!
Joel Grus
Title: Reproducibility as a Vehicle for Engineering Best Practices
Abstract:
As an engineer, I'm a strong advocate for software engineering best practices; and as someone who works in AI, I'm a strong advocate for reproducible research. Fortunately for me, these advocacies overlap, as it turns out that good engineering practices make reproducibility much easier.
In this talk I'll discuss the various reasons reproducibility is important, demonstrate how good engineering discipline leads to more reproducible science, and give some examples from my own work.
Pablo Samuel Castro
Title: Dopamine: thoughts on reproducibility
Abstract:
Last summer we released Dopamine, a library for flexible reinforcement learning research, which has been well received by the community. We built the design of the framework aiming for simplicity, legibility, and flexibility, to enable fast prototyping of new ideas. We believe these design choices are largely responsible for the reception our framework has received. In this talk I will reflect on some of these design choices and how they can help address some of the reproducibility issues that have recently been raised. I will frame these discussions within the context of my academic career, which includes an extended hiatus at the time when the deep learning revolution began!
Avital Oliver
Title: How can reproducibility support understanding?
Abstract:
Let's talk about research code. In this talk, I will compare the constraints in which those are developed with a different way to reproduce a paper -- namely Colab notebooks that can be easily run, explored and shared. These notebooks allow for rich structure unique fit for reproducing papers in a way that facilitates /understanding/. I will propose looking at Depth First Learning, a new ML pedagogical system as a hub for Colab notebooks building up in steps towards important ML papers.
Dhruv Madeka
Title: On reproducible research for large scale production deployments
Soumith Chintala
Title: A checklist for open-sourcing your code for reproducibility and extensibility
Abstract: In this talk, we shall aim to develop a checklist that helps you release your paper, code and data in a way that a maximal number of people can use it. We shall discuss constraints and perspectives of people that you as the author often miss, and to think and improve your release keeping these perspectives in mind.
Jessica Forde
Title: Promoting Science in Machine Learning Research
Abstract: Some researchers have called for increased scientific scrutiny of developments in the field of machine learning, emphasizing the need for advancements in theory, reproducibility, and generalization. Others have observed that successful engineering projects have inspired theoretical understanding. This talk examines how the research community can improve their engineering practices to drive further advances in the field. It demonstrates how we can make full use of open source to build and share experiments to promote mutual understanding and facilitate further research. It examines how we can borrow best practices from other fields of science (psychology, clinical medicine,high energy physics) to further enhance research practices in the community. Work produced in conjunction with researchers at Project Jupyter, FAIR MPK, Harvard, and Mt. Sinai Hospital.
TBD
[1] Lucic, Mario, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. "Are GANs Created Equal? A Large-Scale Study." arXiv preprint arXiv:1711.10337 (2017).
[2] Melis, Gábor, Chris Dyer, and Phil Blunsom. "On the state of the art of evaluation in neural language models." arXiv preprint arXiv:1707.05589 (2017).
[3] Henderson, Peter, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. "Deep reinforcement learning that matters." arXiv preprint arXiv:1709.06560 (2017).
[4] Nie, Xinkun, Xiaoying Tian, Jonathan Taylor, and James Zou. "Why adaptively collected data have negative bias and how to correct for it." (2017).