The goal of this event is to share the evolution of research ideas through specific examples of negative results, retrospectives, and project post-mortems.

Computer systems and machine learning research is often driven by empirical results; improving efficiency and pushing the boundaries of the state of the art are essential goals that are continually furthered by the vetting and discussion of published academic work. However, we observe and experience that reflection, intermediate findings, and negative results are often quietly shelved in this process, despite the educational, scientific, and personal value in airing such experiences. Given the lack of emphasis on negative results, important lessons learned and reflections are neither captured nor maintained by our research communities, further exacerbating the problem.

To this end, we aim to establish a workshop venue centered on reflective and in-depth conversations on the meandering path towards research publications, the path that science is inherently all about: iterating over failures to arrive at a more robust understanding of the world. This is a collaboration inspired by the spirit of NOPE from the Systems/Architecture community and ML-RSA from the ML community.

JOURNE will combine invited talks from prominent ML and Systems researchers on the evolution of and reflection on research trends with specific contributed examples of negative results, retrospectives, and project post-mortems in the MLSys community. We will complement this programming with opportunities for candid discussion and constructive brainstorming about how and when these reflections, intermediate findings, missteps, and negative results are useful for the research community and how they can be supported and brought to light. Our goal is to bring the fundamental principles of scientific research back to the forefront.

What do you do as a researcher when faced with negative results, or when you come upon an unexpected limitation at any stage of a research project? Though it is highly likely you've come across a finding or point of discussion that would help the research community, those morsels may be left behind in preparing a typical academic publication. Similarly, submitted projects rarely wrap-up neatly -- what are the loose ends, reflections, or flaws in a previous work that you'd like to revisit?

JOURNE offers a venue to reveal, discuss, and reflect on previous research projects and experiences. We seek historical perspectives, "bad results", and lessons learned from research ideas across Systems and ML. Contributions may, for example, address the following topics:

• Start-to-finish examination of completed projects, specifically to dissect dead-ends and unworkable solutions encountered along the way.

• Descriptions of challenges in transitioning or translating research ideas to open-source tools or industry applications.

• Evaluations of and reflections on failed projects which uncover and characterize the root cause.

• Design space explorations or comprehensive experiments which suggest a particular technique is unlikely to work.

• Research which uncovers fundamental limitations in scalability, performance, accuracy, or other quantifiable metrics.

• Any research which serves to share the lessons of failure to the broader community, such that we can avoid repeating them in the future.

• Reflections and historical perspectives on specific ideas, general trends, and “techno-social” factors in systems and machine learning research.

In order to encourage broader participation and foster discussion across the MLSys community, submissions do not have to follow a fixed format. Submissions can be in the form of an article, blog, or short paper (~2 pages). Submissions do not need to be anonymized.

Please send your submissions to journe.workshop@gmail.com by March 19, 2021. Notifications for the workshop will be sent on March 26, 2021 with JOURNE being held virtually on April 9, 2021.

Commitment to Diversity

Success of JOURNE depends on sharing varied experiences, retrospectives, and negative outcomes found throughout research. As such, diversity will be central to the success of JOURNE. Keeping that in mind, we see the following vectors of diversity as being critical to the discussions at the workshop and through the programme structure, call for submissions, organization committee, sharing of results, and medium of participation, and we will actively pursue these goals as follows:

• Technical: From a technical standpoint, we will be bringing in experiences from a multitude of subfields, specifically, through a collision of experiences from the ML and Sys community.

• Regional: From a regional standpoint, we will be actively seeking speakers, submissions, and participants from as-of-yet underrepresented regions like MENA, West Asia, Eastern Europe, Latin America, and Oceania.

• Seniority: We also believe that this scientific journey will vary in what it looks like over different levels of experience and thus, we will actively bring together people who are early-career, mid-career, and veterans in the field so that there can be cross-pollination of ideas.

• Ethnic backgrounds and gender: We also recognize that there are many vectors for diversity, and will adopt best practices from workshops like Resistance AI, Black in AI, among others to bring together diversity that extends beyond just gender and race.