NeurIPS 2021 Workshop on

Distribution Shifts

Connecting Methods and Applications

Monday, December 13, 2021

NeurIPS virtual site

Accepted papers

Distribution shiftswhere a model is deployed on a data distribution different from what it was trained onpose significant robustness challenges in real-world ML applications. Such shifts are often unavoidable in the wild and have been shown to substantially degrade model performance in applications such as biomedicine, wildlife conservation, sustainable development, robotics, education, and criminal justice. For example, models can systematically fail when tested on patients from different hospitals or people from different demographics.

This workshop aims to convene a diverse set of domain experts and methods-oriented researchers working on distribution shifts. We are broadly interested in methods, evaluations and benchmarks, and theory for distribution shifts, and we are especially interested in work on distribution shifts that arise naturally in real-world application contexts. Examples of relevant topics include, but are not limited to:

  • Examples of real-world distribution shifts in various application areas. We especially welcome applications that are not widely discussed in the ML research community, e.g., education, sustainable development, and conservation. We encourage submissions that characterize distribution shifts and their effects in real-world applications; it is not at all necessary to propose a solution that is algorithmically novel.

  • Methods for improving robustness to distribution shifts. Relevant settings include domain generalization, domain adaptation, and subpopulation shifts, and we are interested in a wide range of approaches, from uncertainty estimation to causal inference to active data collection. We welcome methods that can work across a variety of shifts, as well as more domain-specific methods that incorporate prior knowledge on the types of shifts we wish to be robust on. We encourage evaluating these methods on real-world distribution shifts.

  • Empirical and theoretical characterization of distribution shifts. Distribution shifts can vary widely in the way in which the data distribution changes, as well as the empirical trends they exhibit. What empirical trends do we observe? What empirical or theoretical frameworks can we use to characterize these different types of shifts and their effects? What kinds of theoretical settings capture useful components of real-world distribution shifts?

  • Benchmarks and evaluations. We especially welcome contributions for subpopulation shifts, as they are underrepresented in current ML benchmarks. We are also interested in evaluation protocols that move beyond the standard assumption of fixed training and test splits -- for which applications would we need to consider other forms of shifts, such as streams of continually-changing data or feedback loops between models and data?

If you have any questions, please contact us at


The workshop will be held on Monday, December 13, and the schedule below is in Pacific Time.

Talks and panels will be hosted on the NeurIPS virtual site:

Links to accepted papers and their posters are also on the virtual site. All attendees will need to register for NeurIPS.

[9:00 - 9:10] Opening remarks

Shiori Sagawa, Stanford University

[9:10 - 9:35] Invited talk: Distribution shifts in AI for social good

Ernest Mwebaze, Makerere University and Sunbird AI

Ernest Mwebaze obtained his doctorate in machine learning from the University of Groningen. He has over 10 years experience in academia where he was part of the faculty at the School of Computing and Informatics Technology of Makerere University in Uganda. At Makerere University he co-led the Makerere Artificial Intelligence research lab and headed several research projects. He has worked with the UN at the Pulse Lab Kampala and with Google AI, in Accra, Ghana. His current portfolio includes being the Strategy Lead at Sunbird AI, a non-profit focused on productization of AI for social good.

[9:35 - 10:00] Invited talk: Dataset Shift in Medicine

Suchi Saria, Johns Hopkins University and Bayesian Health

Suchi Saria is the John C. Malone Associate Professor of computer science at the Whiting School of Engineering and of statistics and health policy at the Bloomberg School of Public Health. She directs the Machine Learning and Healthcare Lab and is the founding research director of the Malone Center for Engineering in Healthcare.

[10:00 - 10:25] Invited talk: ML model debugging: A data perspective

Aleksander Mądry, MIT

Aleksander Madry is the Cadence Design Systems Professor of Computing at MIT, leads the MIT Center for Deployable Machine Learning as well as is a faculty co-lead for the MIT AI Policy Forum. His research interests span algorithms, continuous optimization, and understanding machine learning from a robustness and deployability perspectives. Aleksander's work has been recognized with a number of awards, including an NSF CAREER Award, an Alfred P. Sloan Research Fellowship, an ACM Doctoral Dissertation Award Honorable Mention, and Presburger Award. He received his PhD from MIT in 2011 and, prior to joining the MIT faculty, he spent time at Microsoft Research New England and on the faculty of EPFL.

[10:30 - 11:00] Discussion with Aleksander Mądry, Ernest Mwebaze, and Suchi Saria

Moderated by Fanny Yang, ETH Zurich

[11:00 - 11:25] Invited talk: Increasing robustness to distribution shifts by improving design

Elizabeth Tipton, Northwestern University

Elizabeth Tipton is an Associate Professor of Statistics and Faculty Fellow at the Institute for Policy Research at Northwestern University, where she co-directs the Statistics for Evidence-Based Policy and Practice (STEPP) Center. Her research focuses broadly on the development of methods for improved causal generalizations, both through the design and analysis of experiments and through the use of meta-analysis. She is an Associate Editor at the Journal of Educational and Behavioral Statistics, is a member of the Editorial Boards of Psychological Bulletin, Research Synthesis Methods, and Observational Studies, and is a member of the Board of the Society for Research on Educational Effectiveness. Her work has been funded by the Institute for Education Sciences, the National Science Foundation, the Spencer Foundation and the Raikes Foundation. In 2020, she received the Frederick Mosteller Award for Distinctive Contributions to Systematic Reviewing from the Campbell Collaboration. She previously received Early Career Awards from the American Education Research Association, the Society for Research Synthesis Methods, and the American Psychological Association. Tipton received a Ph.D. in Statistics from Northwestern in 2011.

[11:25 - 11:50] Invited talk: Statistical testing under distribution shifts

Jonas Peters, University of Copenhagen

Jonas is professor in statistics at the Department of Mathematical Sciences at the University of Copenhagen. Previously, he has been a group leader at the Max-Planck-Institute for Intelligent Systems in Tuebingen and a Marie Curie fellow at the Seminar for Statistics, ETH Zurich. He studied Mathematics at the University of Heidelberg and the University of Cambridge and obtained his PhD jointly from MPI and ETH. He is interested in inferring causal relationships from different types of data and in building statistical methods that are robust with respect to distributional shifts. In his research, Jonas seeks to combine theory, methodology, and applications. His work relates to areas such as computational statistics, causal inference, graphical models, independence testing or high-dimensional statistics.

[11:50 - 12:10] Discussion with Elizabeth Tipton and Jonas Peters

Moderated by Hongseok Namkoong, Columbia University

[12:10 - 13:00] Spotlight talks

Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks

Neil Band, Tim G. J. Rudner, Qixuan Feng, Angelos Filos, Zachary Nado, Mike Dusenberry, Ghassen Jerfel, Dustin Tran, Yarin Gal

A Closer Look at Distribution Shifts and Out-of-Distribution Generalization on Graphs

Mucong Ding, Kezhi Kong, Jiuhai Chen, John Kirchenbauer, Micah Goldblum, David P Wipf, Furong Huang, Tom Goldstein

On Adaptivity and Confounding in Contextual Bandit Experiments

Chao Qin, Daniel Russo

Is Importance Weighting Incompatible with Interpolating Classifiers?

Ke Alexander Wang, Niladri Chatterji, Saminul Haque, Tatsunori Hashimoto

[15:00 - 15:50] Break

Join us in the GatherTown lounge at

[15:50 - 16:15] Invited talk: Importance weighting for transfer learning

Masashi Sugiyama, RIKEN and the University of Tokyo

Masashi Sugiyama is Professor at the University of Tokyo and Director of RIKEN Center for Advanced Intelligence Project, Japan. He obtained Doctor of Engineering in Computer Science from Tokyo Institute of Technology. His research interest includes the development and analysis of machine learning algorithms, with particular focus on transfer learning, weakly supervised learning, and noise-robust learning. He coauthored several monographs such as "Machine Learning in Non-Stationary Environments" (MIT Press, 2012), and "Machine Learning from Weak Supervision" (MIT Press, to appear).

[16:15 - 16:40] Invited talk: Robustness through the lens of invariance

Chelsea Finn, Stanford University

Chelsea Finn is an Assistant Professor in Computer Science and Electrical Engineering at Stanford University, where she directs the IRIS lab. Her research interests lie in the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction. Her work has developed meta-learning algorithms that can enable fast few-shot adaptation, algorithms for end-to-end training of robotic perception and control, and self-supervised methods for sensorimotor perception, prediction, and control. She received her Bachelor's degree in Electrical Engineering and Computer Science at MIT and her PhD in Computer Science at UC Berkeley. Her research has been recognized through the ACM doctoral dissertation award, the ONR YIP award, and the MIT Technology Review 35 under 35 Award, and her work has been covered by various media outlets, including the New York Times, Wired, and Bloomberg.

[16:40 - 17:00] Discussion with Masashi Sugiyama and Chelsea Finn

Moderated by Hongseok Namkoong, Columbia University

[17:00 - 18:00] Panel: Future directions for tackling distribution shifts

Moderated by Pang Wei Koh, Stanford University


Andrew Beck, PathAI

Andy earned his MD from Brown Medical School and completed residency and fellowship training in Anatomic Pathology and Molecular Genetic Pathology from Stanford University. He completed a PhD in Biomedical Informatics from Stanford University, where he developed a machine-learning based approach for cancer pathology. He's been certified by the American Board of Pathology in Anatomic Pathology and Molecular Genetic Pathology. Prior to co-founding PathAI, he was on the faculty of Harvard Medical School in the Department of Pathology at Beth Israel Deaconess Medical Center. He has published widely in the fields of cancer biology, cancer pathology, and biomedical informatics.

Jamie Morgenstern, University of Washington

Jamie Morgenstern is an assistant professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. She was previously an assistant professor in the School of Computer Science at Georgia Tech. Prior to starting as faculty, she was fortunate to be hosted by Michael Kearns, Aaron Roth, and Rakesh Vohra as a Warren Center fellow at the University of Pennsylvania. She completed her PhD working with Avrim Blum at Carnegie Mellon University. Her research is on the social impact of machine learning and the impact of social behavior on ML's guarantees.

Judy Hoffman, Georgia Tech

Dr. Judy Hoffman is an Assistant Professor in the School of Interactive Computing at Georgia Tech, a member of the Machine Learning Center, and a Diversity and Inclusion Fellow. Her research lies at the intersection of computer vision and machine learning with specialization in domain adaptation, transfer learning, adversarial robustness, and algorithmic fairness. She has received numerous awards including the Samsung AI Researcher of the Year Award (2021), the NVIDIA female leader in computer vision award (2020), AIMiner top 100 most influential scholars in Machine Learning (2020), MIT EECS Rising Star in 2015, and is a recipient of the NSF Graduate Fellowship. In addition to her research, she co-founded and continues to advise for Women in Computer Vision, an organization which provides mentorship and travel support for early-career women in the computer vision community. Prior to joining Georgia Tech, she was a Research Scientist at Facebook AI Research. She received her PhD in Electrical Engineering and Computer Science from UC Berkeley in 2016 after which she completed Postdocs at Stanford University (2017) and UC Berkeley (2018).

Tatsunori Hashimoto, Stanford University

Tatsunori Hashimoto is an Assistant Professor in the Computer Science Department at Stanford University. He is a member of the statistical machine learning and natural language processing groups at Stanford, and his research uses tools from statistics to make machine learning systems more robust and reliable — especially in challenging tasks involving natural language generation. His work has received the best paper runner-up at the International Conference on Machine Learning and the best paper at the Neural Information Processing Systems workshop on Networks. Before becoming an Assistant Professor, he was a postdoctoral researcher at Stanford with Percy Liang and John Duchi and received his Ph.D. from MIT under the supervision of Tommi Jaakkola and David Gifford.


Stanford University

Stanford University

ETH Zurich

Columbia University

National University of Singapore

Boston University

Stanford University


UC Berkeley

Program Committee

Alexander Robey

Alexandru Tifrea

Ali Taylan Cemgil

Amartya Sanyal

Amita Kamath

Amy X. Lu

Ananya Kumar

Andrew Ilyas

Andrii Zadaianchuk

Apostolos Modas

Aurick Zhou

Byol Kim

David Krueger

David Madras

Dimitris Tsipras

Dina Bashkirova

Donghyun Kim

Eleni Triantafillou

Elliot Creager

Emma Pierson

Eric Wallace

Eric Wong

Erik Jones

Gabriel Ilharco

Guillaume Wang

Haoran Zhang

Henrik Marklund

Irena Gao

Irene Y. Chen

Ishaan Gulrajani

Jacob Clarysse

Jeremiah Zhe Liu

Jie Ren

Jiefeng Chen

John Hewitt

Johnny Wei

Jwala Dhamala

Kai Yuanqing Xiao

Kalpesh Krishna

Kamyar Azizzadenesheli

Karthyek Murthy

Kartik Ahuja

Kibok Lee

Konstantin Donhauser

Kuniaki Saito

Logan Engstrom

Maksym Andriushchenko

Marvin Mengxin Zhang

Meng-Jiun Chiou

Michael Aerni

Michael Oberst

Michael Zhang

Nelson F. Liu

Nicholas R Galbraith

Nicolò Ruggeri

Nimit Sharad Sohoni

Olivia Wiles

Peter Hase

Prithvijit Chattopadhyay

Robin Jia

Rohan Taori

Saeid Asgari

Samarth Mishra

Sang Michael Xie

Sara Beery

Saurabh Garg

Shalmali Joshi

Shibani Santurkar

Shikhar Murty

Siddharth Mysore

Stephan Rabanser

Steve Yadlowsky

Sunil Thulasidasan

Thao Nguyen

Viraj Uday Prabhu

Weihua Hu

Xiang Lisa Li

Ximei Wang

Xinyang Chen

Yang Li

Yash Sharma

Yu-Jie Zhang

Yunbei Xu

Zhangjie Cao

Zhongyi Pei