April 9, 2021
JOURNE
Journal of Opportunities, Unexpected limitations, Retrospectives, Negative results, and Experiences
Workshop at MLSys 2021
The goal of this event is to share the evolution of research ideas through specific examples of negative results, retrospectives, and project post-mortems.
Computer systems and machine learning research is often driven by empirical results; improving efficiency and pushing the boundaries of the state of the art are essential goals that are continually furthered by the vetting and discussion of published academic work. However, we observe and experience that reflection, intermediate findings, and negative results are often quietly shelved in this process, despite the educational, scientific, and personal value in airing such experiences. Given the lack of emphasis on negative results, important lessons learned and reflections are neither captured nor maintained by our research communities, further exacerbating the problem.
To this end, we aim to establish a workshop venue centered on reflective and in-depth conversations on the meandering path towards research publications, the path that science is inherently all about: iterating over failures to arrive at a more robust understanding of the world. This is a collaboration inspired by the spirit of NOPE from the Systems/Architecture community and ML-RSA from the ML community.
JOURNE will combine invited talks from prominent ML and Systems researchers on the evolution of and reflection on research trends with specific contributed examples of negative results, retrospectives, and project post-mortems in the MLSys community. We will complement this programming with opportunities for candid discussion and constructive brainstorming about how and when these reflections, intermediate findings, missteps, and negative results are useful for the research community and how they can be supported and brought to light. Our goal is to bring the fundamental principles of scientific research back to the forefront.
Origins of the workshop
JOURNE is born of two independent but similarly-spirited efforts from the Systems (NOPE) and ML (ML Retrospectives) communities. We jointly acknowledge that our research communities have a bias for publishing “good results'' and lack venues for reflection, critical discussion, and exposure of the shortcomings of published works. Both NOPE and ML Retrospectives have successfully endeavored to provide a platform to learn from the process of how ideas formed. NOPE has been previously hosted at MICRO-2015, MICRO-2016, MICRO-2017, and ASPLOS-2019; ML Retrospectives have been previously hosted at NeurIPS 2019, ICML 2020, and NeurIPS 2020. JOURNE will offer a new space to have expository and productive conversations in the MLSys Community.
The Speakers
Vijay Janapa Reddi
Harvard University
Shalini De Mello
NVIDIA
Luis Ceze
University of Washington
Deborah Raji
Mozilla Foundation
The Panelists
Michael Carbin
MIT
Zachary Lipton
Carnegie Mellon University
Devi Parikh
Georgia Tech / FAIR
Gennady Pekhimenko
University of Toronto
Programme
Time
Speakers
Title
11:00am ET - 11:15am ET
JOURNE Organizers
Welcome to JOURNE
11:15am ET - 12:00pm ET
Luis Ceze
(University of Washington)
Thoughts on Research, Community and Impact
12:00pm ET - 12:45pm ET
Deb Raji
Coming soon
12:45pm ET - 1:00pm ET
Coffee Break
1:00pm ET - 1:45pm ET
Vijay Janapa Reddi
(Harvard University)
The Future of ML is Tiny and Bright: Challenges and Opportunities
1:45pm ET - 2:30pm ET
Shalini De Mello
(NVIDIA)
Bringing your Research Ideas to Life in Real-world Products
2:30pm ET - 3:30pm ET
Coffee/Lunch Break
3:30pm ET - 4:30pm ET
Michael Carbin (MIT)
Zachary Lipton (Carnegie Mellon University)
Devi Parikh (Georgia Tech/FAIR)
Gennady Pekhimenko (University of Toronto)
Industry / Academia panel
4:30pm ET - 5:30pm ET
Contributed talks / brainstorming session
5:30pm ET - 5:45pm ET
JOURNE Organizers
Concluding remarks
Call for Contributions
What do you do as a researcher when faced with negative results, or when you come upon an unexpected limitation at any stage of a research project? Though it is highly likely you've come across a finding or point of discussion that would help the research community, those morsels may be left behind in preparing a typical academic publication. Similarly, submitted projects rarely wrap-up neatly -- what are the loose ends, reflections, or flaws in a previous work that you'd like to revisit?
JOURNE offers a venue to reveal, discuss, and reflect on previous research projects and experiences. We seek historical perspectives, "bad results", and lessons learned from research ideas across Systems and ML. Contributions may, for example, address the following topics:
Start-to-finish examination of completed projects, specifically to dissect dead-ends and unworkable solutions encountered along the way.
Descriptions of challenges in transitioning or translating research ideas to open-source tools or industry applications.
Evaluations of and reflections on failed projects which uncover and characterize the root cause.
Design space explorations or comprehensive experiments which suggest a particular technique is unlikely to work.
Research which uncovers fundamental limitations in scalability, performance, accuracy, or other quantifiable metrics.
Any research which serves to share the lessons of failure to the broader community, such that we can avoid repeating them in the future.
Reflections and historical perspectives on specific ideas, general trends, and “techno-social” factors in systems and machine learning research.
In order to encourage broader participation and foster discussion across the MLSys community, submissions do not have to follow a fixed format. Submissions can be in the form of an article, blog, or short paper (~2 pages). Submissions do not need to be anonymized.
Please send your submissions to journe.workshop@gmail.com by March 19, 2021. Notifications for the workshop will be sent on March 26, 2021 with JOURNE being held virtually on April 9, 2021.
The Organizers
Abhishek Gupta
Montreal AI Ethics Institute / Microsoft
Udit Gupta
Harvard University/FAIR
Mayoore Jaiswal
Nvidia
Lillian Pentecost
Harvard University
Shagun Sodhani
FAIR
David Brooks
Harvard University
Joelle Pineau
McGill University/FAIR
Commitment to Diversity
Success of JOURNE depends on sharing varied experiences, retrospectives, and negative outcomes found throughout research. As such, diversity will be central to the success of JOURNE. Keeping that in mind, we see the following vectors of diversity as being critical to the discussions at the workshop and through the programme structure, call for submissions, organization committee, sharing of results, and medium of participation, and we will actively pursue these goals as follows:
Technical: From a technical standpoint, we will be bringing in experiences from a multitude of subfields, specifically, through a collision of experiences from the ML and Sys community.
Regional: From a regional standpoint, we will be actively seeking speakers, submissions, and participants from as-of-yet underrepresented regions like MENA, West Asia, Eastern Europe, Latin America, and Oceania.
Seniority: We also believe that this scientific journey will vary in what it looks like over different levels of experience and thus, we will actively bring together people who are early-career, mid-career, and veterans in the field so that there can be cross-pollination of ideas.
Ethnic backgrounds and gender: We also recognize that there are many vectors for diversity, and will adopt best practices from workshops like Resistance AI, Black in AI, among others to bring together diversity that extends beyond just gender and race.