NeurIPS 2021 Workshop on
Distribution Shifts
Connecting Methods and Applications
Distribution shifts—where a model is deployed on a data distribution different from what it was trained on—pose significant robustness challenges in real-world ML applications. Such shifts are often unavoidable in the wild and have been shown to substantially degrade model performance in applications such as biomedicine, wildlife conservation, sustainable development, robotics, education, and criminal justice. For example, models can systematically fail when tested on patients from different hospitals or people from different demographics.
This workshop aims to convene a diverse set of domain experts and methods-oriented researchers working on distribution shifts. We are broadly interested in methods, evaluations and benchmarks, and theory for distribution shifts, and we are especially interested in work on distribution shifts that arise naturally in real-world application contexts. Examples of relevant topics include, but are not limited to:
Examples of real-world distribution shifts in various application areas. We especially welcome applications that are not widely discussed in the ML research community, e.g., education, sustainable development, and conservation. We encourage submissions that characterize distribution shifts and their effects in real-world applications; it is not at all necessary to propose a solution that is algorithmically novel.
Methods for improving robustness to distribution shifts. Relevant settings include domain generalization, domain adaptation, and subpopulation shifts, and we are interested in a wide range of approaches, from uncertainty estimation to causal inference to active data collection. We welcome methods that can work across a variety of shifts, as well as more domain-specific methods that incorporate prior knowledge on the types of shifts we wish to be robust on. We encourage evaluating these methods on real-world distribution shifts.
Empirical and theoretical characterization of distribution shifts. Distribution shifts can vary widely in the way in which the data distribution changes, as well as the empirical trends they exhibit. What empirical trends do we observe? What empirical or theoretical frameworks can we use to characterize these different types of shifts and their effects? What kinds of theoretical settings capture useful components of real-world distribution shifts?
Benchmarks and evaluations. We especially welcome contributions for subpopulation shifts, as they are underrepresented in current ML benchmarks. We are also interested in evaluation protocols that move beyond the standard assumption of fixed training and test splits -- for which applications would we need to consider other forms of shifts, such as streams of continually-changing data or feedback loops between models and data?
If you have any questions, please contact us at distshift-workshop-2021@googlegroups.com.
Schedule
The workshop will be held on Monday, December 13, and the schedule below is in Pacific Time.
Talks and panels will be hosted on the NeurIPS virtual site: https://neurips.cc/virtual/2021/workshop/21859
Links to accepted papers and their posters are also on the virtual site. All attendees will need to register for NeurIPS.
[9:00 - 9:10] Opening remarks
Shiori Sagawa, Stanford University
[9:10 - 9:35] Invited talk: Distribution shifts in AI for social good
Ernest Mwebaze, Makerere University and Sunbird AI
Ernest Mwebaze obtained his doctorate in machine learning from the University of Groningen. He has over 10 years experience in academia where he was part of the faculty at the School of Computing and Informatics Technology of Makerere University in Uganda. At Makerere University he co-led the Makerere Artificial Intelligence research lab and headed several research projects. He has worked with the UN at the Pulse Lab Kampala and with Google AI, in Accra, Ghana. His current portfolio includes being the Strategy Lead at Sunbird AI, a non-profit focused on productization of AI for social good.
[9:35 - 10:00] Invited talk: Dataset Shift in Medicine
Suchi Saria, Johns Hopkins University and Bayesian Health
Suchi Saria is the John C. Malone Associate Professor of computer science at the Whiting School of Engineering and of statistics and health policy at the Bloomberg School of Public Health. She directs the Machine Learning and Healthcare Lab and is the founding research director of the Malone Center for Engineering in Healthcare.
[10:00 - 10:25] Invited talk: ML model debugging: A data perspective
Aleksander Mądry, MIT
Aleksander Madry is the Cadence Design Systems Professor of Computing at MIT, leads the MIT Center for Deployable Machine Learning as well as is a faculty co-lead for the MIT AI Policy Forum. His research interests span algorithms, continuous optimization, and understanding machine learning from a robustness and deployability perspectives. Aleksander's work has been recognized with a number of awards, including an NSF CAREER Award, an Alfred P. Sloan Research Fellowship, an ACM Doctoral Dissertation Award Honorable Mention, and Presburger Award. He received his PhD from MIT in 2011 and, prior to joining the MIT faculty, he spent time at Microsoft Research New England and on the faculty of EPFL.
[10:30 - 11:00] Discussion with Aleksander Mądry, Ernest Mwebaze, and Suchi Saria
Moderated by Fanny Yang, ETH Zurich
[11:00 - 11:25] Invited talk: Increasing robustness to distribution shifts by improving design
Elizabeth Tipton, Northwestern University
Elizabeth Tipton is an Associate Professor of Statistics and Faculty Fellow at the Institute for Policy Research at Northwestern University, where she co-directs the Statistics for Evidence-Based Policy and Practice (STEPP) Center. Her research focuses broadly on the development of methods for improved causal generalizations, both through the design and analysis of experiments and through the use of meta-analysis. She is an Associate Editor at the Journal of Educational and Behavioral Statistics, is a member of the Editorial Boards of Psychological Bulletin, Research Synthesis Methods, and Observational Studies, and is a member of the Board of the Society for Research on Educational Effectiveness. Her work has been funded by the Institute for Education Sciences, the National Science Foundation, the Spencer Foundation and the Raikes Foundation. In 2020, she received the Frederick Mosteller Award for Distinctive Contributions to Systematic Reviewing from the Campbell Collaboration. She previously received Early Career Awards from the American Education Research Association, the Society for Research Synthesis Methods, and the American Psychological Association. Tipton received a Ph.D. in Statistics from Northwestern in 2011.
[11:25 - 11:50] Invited talk: Statistical testing under distribution shifts
Jonas Peters, University of Copenhagen
Jonas is professor in statistics at the Department of Mathematical Sciences at the University of Copenhagen. Previously, he has been a group leader at the Max-Planck-Institute for Intelligent Systems in Tuebingen and a Marie Curie fellow at the Seminar for Statistics, ETH Zurich. He studied Mathematics at the University of Heidelberg and the University of Cambridge and obtained his PhD jointly from MPI and ETH. He is interested in inferring causal relationships from different types of data and in building statistical methods that are robust with respect to distributional shifts. In his research, Jonas seeks to combine theory, methodology, and applications. His work relates to areas such as computational statistics, causal inference, graphical models, independence testing or high-dimensional statistics.
[11:50 - 12:10] Discussion with Elizabeth Tipton and Jonas Peters
Moderated by Hongseok Namkoong, Columbia University
[12:10 - 13:00] Spotlight talks
Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks
Neil Band, Tim G. J. Rudner, Qixuan Feng, Angelos Filos, Zachary Nado, Mike Dusenberry, Ghassen Jerfel, Dustin Tran, Yarin Gal
A Closer Look at Distribution Shifts and Out-of-Distribution Generalization on Graphs
Mucong Ding, Kezhi Kong, Jiuhai Chen, John Kirchenbauer, Micah Goldblum, David P Wipf, Furong Huang, Tom Goldstein
On Adaptivity and Confounding in Contextual Bandit Experiments
Chao Qin, Daniel Russo
Is Importance Weighting Incompatible with Interpolating Classifiers?
Ke Alexander Wang, Niladri Chatterji, Saminul Haque, Tatsunori Hashimoto
[13:00 - 15:00] Poster session
We have three GatherTown rooms for the poster session:
https://eventhosts.gather.town/gEiGKMGAHDYH2sue/distshift-poster-room-1
https://eventhosts.gather.town/p3dsWSj8s2PMlKDV/distshift-poster-room-2
https://eventhosts.gather.town/bFGVxyoHTFr3vJhd/distshift-poster-room-3
Please visit the virtual site for links to individual posters.
[15:00 - 15:50] Break
Join us in the GatherTown lounge at https://eventhosts.gather.town/GwFs3d9SZRNEKct3/distshift-landing-room.
[15:50 - 16:15] Invited talk: Importance weighting for transfer learning
Masashi Sugiyama, RIKEN and the University of Tokyo
Masashi Sugiyama is Professor at the University of Tokyo and Director of RIKEN Center for Advanced Intelligence Project, Japan. He obtained Doctor of Engineering in Computer Science from Tokyo Institute of Technology. His research interest includes the development and analysis of machine learning algorithms, with particular focus on transfer learning, weakly supervised learning, and noise-robust learning. He coauthored several monographs such as "Machine Learning in Non-Stationary Environments" (MIT Press, 2012), and "Machine Learning from Weak Supervision" (MIT Press, to appear).
[16:15 - 16:40] Invited talk: Robustness through the lens of invariance
Chelsea Finn, Stanford University
Chelsea Finn is an Assistant Professor in Computer Science and Electrical Engineering at Stanford University, where she directs the IRIS lab. Her research interests lie in the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction. Her work has developed meta-learning algorithms that can enable fast few-shot adaptation, algorithms for end-to-end training of robotic perception and control, and self-supervised methods for sensorimotor perception, prediction, and control. She received her Bachelor's degree in Electrical Engineering and Computer Science at MIT and her PhD in Computer Science at UC Berkeley. Her research has been recognized through the ACM doctoral dissertation award, the ONR YIP award, and the MIT Technology Review 35 under 35 Award, and her work has been covered by various media outlets, including the New York Times, Wired, and Bloomberg.
[16:40 - 17:00] Discussion with Masashi Sugiyama and Chelsea Finn
Moderated by Hongseok Namkoong, Columbia University
[17:00 - 18:00] Panel: Future directions for tackling distribution shifts
Moderated by Pang Wei Koh, Stanford University
Panelists:
Andy earned his MD from Brown Medical School and completed residency and fellowship training in Anatomic Pathology and Molecular Genetic Pathology from Stanford University. He completed a PhD in Biomedical Informatics from Stanford University, where he developed a machine-learning based approach for cancer pathology. He's been certified by the American Board of Pathology in Anatomic Pathology and Molecular Genetic Pathology. Prior to co-founding PathAI, he was on the faculty of Harvard Medical School in the Department of Pathology at Beth Israel Deaconess Medical Center. He has published widely in the fields of cancer biology, cancer pathology, and biomedical informatics.
Jamie Morgenstern is an assistant professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. She was previously an assistant professor in the School of Computer Science at Georgia Tech. Prior to starting as faculty, she was fortunate to be hosted by Michael Kearns, Aaron Roth, and Rakesh Vohra as a Warren Center fellow at the University of Pennsylvania. She completed her PhD working with Avrim Blum at Carnegie Mellon University. Her research is on the social impact of machine learning and the impact of social behavior on ML's guarantees.
Dr. Judy Hoffman is an Assistant Professor in the School of Interactive Computing at Georgia Tech, a member of the Machine Learning Center, and a Diversity and Inclusion Fellow. Her research lies at the intersection of computer vision and machine learning with specialization in domain adaptation, transfer learning, adversarial robustness, and algorithmic fairness. She has received numerous awards including the Samsung AI Researcher of the Year Award (2021), the NVIDIA female leader in computer vision award (2020), AIMiner top 100 most influential scholars in Machine Learning (2020), MIT EECS Rising Star in 2015, and is a recipient of the NSF Graduate Fellowship. In addition to her research, she co-founded and continues to advise for Women in Computer Vision, an organization which provides mentorship and travel support for early-career women in the computer vision community. Prior to joining Georgia Tech, she was a Research Scientist at Facebook AI Research. She received her PhD in Electrical Engineering and Computer Science from UC Berkeley in 2016 after which she completed Postdocs at Stanford University (2017) and UC Berkeley (2018).
Tatsunori Hashimoto is an Assistant Professor in the Computer Science Department at Stanford University. He is a member of the statistical machine learning and natural language processing groups at Stanford, and his research uses tools from statistics to make machine learning systems more robust and reliable — especially in challenging tasks involving natural language generation. His work has received the best paper runner-up at the International Conference on Machine Learning and the best paper at the Neural Information Processing Systems workshop on Networks. Before becoming an Assistant Professor, he was a postdoctoral researcher at Stanford with Percy Liang and John Duchi and received his Ph.D. from MIT under the supervision of Tommi Jaakkola and David Gifford.
Organizers
Stanford University
Stanford University
ETH Zurich
Columbia University
National University of Singapore
Boston University
Stanford University
Microsoft
UC Berkeley
Program Committee
Alexander Robey
Alexandru Tifrea
Ali Taylan Cemgil
Amartya Sanyal
Amita Kamath
Amy X. Lu
Ananya Kumar
Andrew Ilyas
Andrii Zadaianchuk
Apostolos Modas
Aurick Zhou
Byol Kim
David Krueger
David Madras
Dimitris Tsipras
Dina Bashkirova
Donghyun Kim
Eleni Triantafillou
Elliot Creager
Emma Pierson
Eric Wallace
Eric Wong
Erik Jones
Gabriel Ilharco
Guillaume Wang
Haoran Zhang
Henrik Marklund
Irena Gao
Irene Y. Chen
Ishaan Gulrajani
Jacob Clarysse
Jeremiah Zhe Liu
Jie Ren
Jiefeng Chen
John Hewitt
Johnny Wei
Jwala Dhamala
Kai Yuanqing Xiao
Kalpesh Krishna
Kamyar Azizzadenesheli
Karthyek Murthy
Kartik Ahuja
Kibok Lee
Konstantin Donhauser
Kuniaki Saito
Logan Engstrom
Maksym Andriushchenko
Marvin Mengxin Zhang
Meng-Jiun Chiou
Michael Aerni
Michael Oberst
Michael Zhang
Nelson F. Liu
Nicholas R Galbraith
Nicolò Ruggeri
Nimit Sharad Sohoni
Olivia Wiles
Peter Hase
Prithvijit Chattopadhyay
Robin Jia
Rohan Taori
Saeid Asgari
Samarth Mishra
Sang Michael Xie
Sara Beery
Saurabh Garg
Shalmali Joshi
Shibani Santurkar
Shikhar Murty
Siddharth Mysore
Stephan Rabanser
Steve Yadlowsky
Sunil Thulasidasan
Thao Nguyen
Viraj Uday Prabhu
Weihua Hu
Xiang Lisa Li
Ximei Wang
Xinyang Chen
Yang Li
Yash Sharma
Yu-Jie Zhang
Yunbei Xu
Zhangjie Cao
Zhongyi Pei