April 9, 2021


Journal of Opportunities, Unexpected limitations, Retrospectives, Negative results, and Experiences

Workshop at MLSys 2021

The goal of this event is to share the evolution of research ideas through specific examples of negative results, retrospectives, and project post-mortems.

Computer systems and machine learning research is often driven by empirical results; improving efficiency and pushing the boundaries of the state of the art are essential goals that are continually furthered by the vetting and discussion of published academic work. However, we observe and experience that reflection, intermediate findings, and negative results are often quietly shelved in this process, despite the educational, scientific, and personal value in airing such experiences. Given the lack of emphasis on negative results, important lessons learned and reflections are neither captured nor maintained by our research communities, further exacerbating the problem.

To this end, we aim to establish a workshop venue centered on reflective and in-depth conversations on the meandering path towards research publications, the path that science is inherently all about: iterating over failures to arrive at a more robust understanding of the world. This is a collaboration inspired by the spirit of NOPE from the Systems/Architecture community and ML-RSA from the ML community.

JOURNE will combine invited talks from prominent ML and Systems researchers on the evolution of and reflection on research trends with specific contributed examples of negative results, retrospectives, and project post-mortems in the MLSys community. We will complement this programming with opportunities for candid discussion and constructive brainstorming about how and when these reflections, intermediate findings, missteps, and negative results are useful for the research community and how they can be supported and brought to light. Our goal is to bring the fundamental principles of scientific research back to the forefront.

Origins of the workshop

JOURNE is born of two independent but similarly-spirited efforts from the Systems (NOPE) and ML (ML Retrospectives) communities. We jointly acknowledge that our research communities have a bias for publishing “good results'' and lack venues for reflection, critical discussion, and exposure of the shortcomings of published works. Both NOPE and ML Retrospectives have successfully endeavored to provide a platform to learn from the process of how ideas formed. NOPE has been previously hosted at MICRO-2015, MICRO-2016, MICRO-2017, and ASPLOS-2019; ML Retrospectives have been previously hosted at NeurIPS 2019, ICML 2020, and NeurIPS 2020. JOURNE will offer a new space to have expository and productive conversations in the MLSys Community.

The Speakers

Vijay Janapa Reddi

Harvard University

Shalini De Mello


Luis Ceze

University of Washington

Deborah Raji

Mozilla Foundation

The Panelists

Michael Carbin


Zachary Lipton

Carnegie Mellon University

Devi Parikh

Georgia Tech / FAIR

Gennady Pekhimenko

University of Toronto





11:00am ET - 11:15am ET

JOURNE Organizers

Welcome to JOURNE

11:15am ET - 12:00pm ET

Luis Ceze

(University of Washington)

Thoughts on Research, Community and Impact

12:00pm ET - 12:45pm ET

Deb Raji

Coming soon

12:45pm ET - 1:00pm ET

Coffee Break

1:00pm ET - 1:45pm ET

Vijay Janapa Reddi

(Harvard University)

The Future of ML is Tiny and Bright: Challenges and Opportunities

1:45pm ET - 2:30pm ET

Shalini De Mello


Bringing your Research Ideas to Life in Real-world Products

2:30pm ET - 3:30pm ET

Coffee/Lunch Break

3:30pm ET - 4:30pm ET

Michael Carbin (MIT)

Zachary Lipton (Carnegie Mellon University)

Devi Parikh (Georgia Tech/FAIR)

Gennady Pekhimenko (University of Toronto)

Industry / Academia panel

4:30pm ET - 5:30pm ET

Contributed talks / brainstorming session

5:30pm ET - 5:45pm ET

JOURNE Organizers

Concluding remarks

Call for Contributions

What do you do as a researcher when faced with negative results, or when you come upon an unexpected limitation at any stage of a research project? Though it is highly likely you've come across a finding or point of discussion that would help the research community, those morsels may be left behind in preparing a typical academic publication. Similarly, submitted projects rarely wrap-up neatly -- what are the loose ends, reflections, or flaws in a previous work that you'd like to revisit?

JOURNE offers a venue to reveal, discuss, and reflect on previous research projects and experiences. We seek historical perspectives, "bad results", and lessons learned from research ideas across Systems and ML. Contributions may, for example, address the following topics:

  • Start-to-finish examination of completed projects, specifically to dissect dead-ends and unworkable solutions encountered along the way.

  • Descriptions of challenges in transitioning or translating research ideas to open-source tools or industry applications.

  • Evaluations of and reflections on failed projects which uncover and characterize the root cause.

  • Design space explorations or comprehensive experiments which suggest a particular technique is unlikely to work.

  • Research which uncovers fundamental limitations in scalability, performance, accuracy, or other quantifiable metrics.

  • Any research which serves to share the lessons of failure to the broader community, such that we can avoid repeating them in the future.

  • Reflections and historical perspectives on specific ideas, general trends, and “techno-social” factors in systems and machine learning research.

In order to encourage broader participation and foster discussion across the MLSys community, submissions do not have to follow a fixed format. Submissions can be in the form of an article, blog, or short paper (~2 pages). Submissions do not need to be anonymized.

Please send your submissions to journe.workshop@gmail.com by March 19, 2021. Notifications for the workshop will be sent on March 26, 2021 with JOURNE being held virtually on April 9, 2021.

The Organizers

Abhishek Gupta

Montreal AI Ethics Institute / Microsoft

Udit Gupta

Harvard University/FAIR

Mayoore Jaiswal


Lillian Pentecost

Harvard University

Shagun Sodhani


David Brooks

Harvard University

Joelle Pineau

McGill University/FAIR

Commitment to Diversity

Success of JOURNE depends on sharing varied experiences, retrospectives, and negative outcomes found throughout research. As such, diversity will be central to the success of JOURNE. Keeping that in mind, we see the following vectors of diversity as being critical to the discussions at the workshop and through the programme structure, call for submissions, organization committee, sharing of results, and medium of participation, and we will actively pursue these goals as follows:

  • Technical: From a technical standpoint, we will be bringing in experiences from a multitude of subfields, specifically, through a collision of experiences from the ML and Sys community.

  • Regional: From a regional standpoint, we will be actively seeking speakers, submissions, and participants from as-of-yet underrepresented regions like MENA, West Asia, Eastern Europe, Latin America, and Oceania.

  • Seniority: We also believe that this scientific journey will vary in what it looks like over different levels of experience and thus, we will actively bring together people who are early-career, mid-career, and veterans in the field so that there can be cross-pollination of ideas.

  • Ethnic backgrounds and gender: We also recognize that there are many vectors for diversity, and will adopt best practices from workshops like Resistance AI, Black in AI, among others to bring together diversity that extends beyond just gender and race.