References

General

• Practical Uncertainty Estimation and Out-of-Distribution Robustness in Deep Learning. NeurIPS 2020 Tutorial.

• A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. 2021.

• Bayesian Deep Learning and a Probabilistic Perspective of Model Construction. ICML 2020 Tutorial.

• Probability Calibration Workshop. PyData 2020.

• Uncertainty Estimation and Bayesian Neural Networks. PyData 2018.

• Symposium on Conformal and Probabilistic Prediction with Applications. COPA conference 2012-2022.

• Workshop on Distribution-Free Uncertainty Quantification. DFUQ @ ICML 2021-2022.

• Uncertainty & Robustness in Deep Learning Workshop. UDL @ ICML 2022.

Probability calibration

• Nie et al. "What Can We Learn from Collective Human Opinions on Natural Language Inference Data?". EMNLP 2020.

• Desai & Durrett "Calibration of Pre-trained Transformers". EMNLP 2020.

• Kuleshov & Liang "Calibrated Structured Prediction". NeurIPS 2015.

• Kull et al. "Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration". Neurips 2019.

• Kumar & Sarawagi. "Calibration of Encoder-Decoder Models for Neural Machine Translation". 2019.

• Zhao et. al. "Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration". 2021.

• Guo et al. "On Calibration of Modern Neural Networks". ICML 2017.

• Zadrozny & Elkan, "Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers". ICML 2001.

• Naeini et al. "Obtaining Well Calibrated Probabilities Using Bayesian Binning". AAAI 2015.

• Kumar et. al. "Verified Uncertainty Calibration". NeurIPS 2020.

• Mielke et al. "Reducing conversational agents' overconfidence through linguistic calibration". TACL 2022.

• Kadavath et al. "Language Models (Mostly) Know What They Know". 2022.

• Lin et al. "Teaching Models to Express Their Uncertainty in Words". 2022.

• Jiang et al. "How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering". TACL 2021.

• Liu et al. "BRIO: Bringing Order to Abstractive Summarization". ACL 2022

• Zhang et al. "Knowing More About Questions Can Help: Improving Calibration in Question Answering". Findings ACL 2021

Bayesian approaches

• Tran et al. "Practical Uncertainty Estimation & Out-of-Distribution Robustness in Deep Learning." NeurIPS Tutorial, 2020.

• Gal and Ghahramani. "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning." ICML 2016.

• Houlsby et al. "Bayesian Active Learning for Classification and Preference Learning." 2011.

• Siddhant and Lipton. "Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study." EMNLP 2018.

• Lakshminarayanan et al. "Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles." NeurIPS 2017.

• Xiao & Wang "Quantifying Uncertainties in Natural Language Processing Tasks". AAAI 2019.

• Li et al. "Deep Bayesian Gaussian processes for uncertainty estimation in electronic health records". Nature Scientific Reports 2021.

Conformal Prediction

• Vovk et al. "Algorithmic Learning in a Random World". 2005.

• Angelopoulos et al. “Uncertainty Sets for Image Classifiers using Conformal Prediction”. ICLR 2021

• Maltoudoglou et al. “BERT-based conformal predictor for sentiment analysis”. CPPA 2020

• Dey et al. “Conformal prediction for text infilling and part-of-speech prediction”. 2021

• Fisch et al. “Efficient Conformal Prediction via Cascaded Inference with Expanded Admission”. ICLR 2021

• Schuster et al. “Consistent Accelerated Inference via Confident Adaptive Transformers”. EMNLP 2021

• Angelopoulos et al. “Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control“. 2021

• Schuster et al. “Confident Adaptive Language Modeling”. NeurIPS 2022

• Angelopoulos et al. “Conformal Risk Control“. 2022

• Tibshirani et al. “Conformal Prediction Under Covariate Shift”. 2020

• Card et al. "Deep Weighted Averaging Classifiers". FAT* 2019

• Bulian et al. “Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation”. EMNLP 2022

Selective prediction and out-of-domain detection

• El-Yaniv and Weiner. "On the foundations of noise-free selective classification." JMLR 2010.

• Hendrycks and Gimpel. “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks.” ICLR 2017.

• Rodriguez, Feng, Iyyer, He, and Boyd-Graber. “Quizbowl: The Case for Incremental Question Answering.” arxiv 2019.

• Kamath, Jia, and Liang. “Selective Question Answering under Domain Shift.” ACL 2020.

• Zhang, Gong, and Choi. “Knowing More About Questions Can Help: Improving Calibration in Question Answering.” ACL Findings 2021.

• Hendrycks, Mazeika, and Dietterich. “Deep Anomaly Detection with Outlier Exposure.” ICLR 2019.

• Hendrycks, Liu, Wallace, Dziedzic, Krishnan, and Song. “Pretrained Transformers Improve Out-of-Distribution Robustness.” ACL 2020.

• Zhou, Liu, and Chen. “Contrastive Out-of-Distribution Detection for Pretrained Transformers.” EMNLP 2021.

• Arora, Huang, and He. “Types of Out-of-Distribution Texts and How to Detect Them.” EMNLP 2021.

• Garg & Moschitti. Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering. EMNLP 2021.