Uncertainty Quantification
for Text Classification

Available upon request.

• CLINC150

The following tutorials and notebooks provide example methods for Deep Learning uncertainty. The methods are implemented with Tensorflow and inspired by TF tutorials and other public libraries (e.g., Google's Uncertainty Baselines).

•  SNGP model using toy 2D image data (i.e., scikit-learn's two moons dataset). See this TF tutorial.

•  BERT-SNGP model, where SNGP is built on top of a BERT encoder to improve the model's ability in detecting out-of-scope queries.
  The CLINC150 dataset is used. See this Colab notebook, which follows closely the TF tutorial and adds additional uncertainty quantification metrics for out-of-domain detection as well as in-domain evaluation.

• Deterministic TextCNN model, which has been inspired by Google's Uncertainty Baselines library, see here. We train a TextCNN model on the CLINC150 dataset as well as a Deep Ensemble of TextCNN models and evaluate their OOD performance (see this Colab notebook).

• Other related materials:

  • Jordy Van Landeghem's uncertainty-bench repository on Github.

  • Liudmila Prokhorenkova's tutorial on uncertainty estimation with CatBoost with this Jupyter notebook.

  • Murat Sensoy's original Colab notebook on Evidential Neural Network.

• Benchmarking Scalable Predictive Uncertainty in Text Classification

• Epistemic Neural Networks

• Uncertainty in Gradient Boosting via Ensembles

• Evidential Deep Learning to Quantify Classification Uncertainty

• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation

• A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness

• Language Models (Mostly) Know What They Know

• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering

• Teaching Models to Express Their Uncertainty in Words

• Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

• Distinguishability Calibration to In-Context Learning

Thursday, 6th April 2023. Half-Day (Afternoon).
Swift Suite, Radisson Blu Royal Hotel, Golden Lane, Dublin 2, D08 VRR7, Ireland

• 1:30pm-3:00pm Session 1/2

• 3:00pm-3:30pm Coffee Break

• 3:30pm-5:00pm Session 2/2

Sunday, 23rd July 2023. Full-Day (Online Only).
    Time Zone: CST (GMT+8)

• 10:00PM-11:30PM  Session 1/4 DZ

• 11:30PM-12:00AM  Short Break

• 12:00AM-01:30AM  Session 2/4 MS

• 01:30AM-02:30AM  Long Break

• 02:30AM-04:00AM  Session 3/4 DZ

• 04:00AM-04:30AM  Short Break

• 04:30AM-06:00AM  Session 4/4 YH+LG

[NOTICE] Due to an unexpected technical problem with the EventX platform, the live streaming event of our SIGIR'23 Virtual Tutorial has been cancelled. We are really sorry for the inconvenience caused. The tutorial slides and recorded video clips will be provided to the conference participants later.