NeurIPS 2019 Workshop on Information Theory and Machine Learning
Workshop Schedule
9:20-9:30 Introductory Remarks
9:30-10:00 [Invited Talk] Ayfer Özgür: Communication-Efficient Distributed Learning
10:00-10:40 [Invited Talk] Stefano Soatto and Alessandro Achille: Information in the Weights and Emergent Properties of Deep Neural Networks
10:40-11:00 Coffee Break and Poster Setup
11:00-11:15 [Contributed Talk] Rob Brekelmans: Understanding Thermodynamic Variational Inference
11:15-11:30 [Contributed Talk] Bob (Robert) Williamson: Data Processing Equalities
11:30-11:45 [Contributed Talk] Jose Gallego: GAIT: A Geometric Approach to Information Theory
12:00-14:00 Lunch + Poster Setup
14:00-14:30 [Invited Talk] Varun Jog: Adversarial risk via optimal transport and optimal couplings
14:30-15:00 [Invited Talk] Po-Ling Loh: Robust information bottleneck
15:00-15:20 Coffee Break
15:20-15:50 [Invited Talk] Aaron van den Oord: Contrastive Predictive Coding
15:50-16:20 [Invited Talk] Alexander A. Alemi: A Case for Compressed Representations
16:25-17:00 [Poster Spotlight]
17:00-18:00 [Poster Session]
Accepted Papers
Compression without Quantization. Gergely Flamich, Marton Havasi, Jose Miguel Hernandez-Lobato
Guess First to Enable Better Compression and Adversarial Robustness. Sicheng Zhu, Bang An, Shiyu Niu
Multilabel prediction in log time and data-dependent grouping. Shashanka Ubaru, Sanjeeb Dash, Oktay Gunluk, Arya Mazumdar
Estimating mutual information using repeated measures. Charles Zheng, Francisco Pereira, Yuval Benjamini
On Mutual Information Maximization for Representation Learning. Michael Tschannen, Josip Djolonga, Paul Rubenstein, Sylvain Gelly, Mario Lucic
Analysis of Deep Neural Networks using Tensor Kernels and Matrix--Based Renyi's Entropy. Kristoffer Wickstrom
Entropic Graph Spectrum. [supplement] Diego Granziol, Binxin Ru, Stefan Zohren, Xiaowen Dong, Michael A. Osborne, Stephen Roberts.
On PAC-Bayes Bounds for Deep Neural Networks using the Loss Curvature. Konstantinos Pitas
Dynamical Coding for Distributed Matrix Multiplication. Xian Su, Xiaodi Fan, Jun Li
Grassmannian Packings in Neural Networks: Learning with Maximal Subspace Packings for Diversity and Anti-Sparsity. Dian Ang Yap, Nicholas Roberts, Vinay Uday Prabhu
Auto-encoding Quantization Model for Short Texts. Xiaobao Wu, Chunping Li, Yan Zhu, Yishu Miao
Reducing overfitting by minimizing label information in weights. Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, Aram Galstyan
Data Processing Equalities. Robert Williamson
Kullback-Leibler Divergence Estimation Using Variationally Weighted Kernel Density Estimators. Sangwoong Yoon, Yung-Kyun Noh, Frank Park
Variational f-divergence Minimization. Mingtian Zhang
InfoMax-GAN: Mutual Information Maximization for Improved Adversarial Image Generation. Kwot Sin Lee, Ngoc-Trung Tran, Ngai-Man Cheung
Quantifying the effect of representations on task complexity. Julian Zilly, Lorenz Hetzel, Andrea Censi, Emilio Frazzoli
Mutual Information heatmaps as a tool for interpretability. Linda Petrini, Tristan Sylvain, R Devon Hjelm
Bounding the Multivariate Mutual Information. Ian Fischer
Using Privileged Information to Improve Prediction in Health Data: A Case Study. Jongoh Jeong, Do Hyung Kwon, Min Joon So, Anita Raja, Shivani Ghatge, Nicolae Lari, Ansaf Salleb Aouissi
On Predictive Information Sub-optimality of RNNs. Zhe Dong, Deniz Oktay, Ben Poole, Alex Alemi
Weakly Supervised Content and Style Disentanglement with Gaussian Mixture VAEs. Jan Nikolas Morshuis, Colin Samplawski, Moin Nabi
Variational Predictive Information Bottleneck. Alex Alemi
Deep Clustering by Gaussian Mixture Variational Autoencoders with Graph Embedding. Linxiao Yang, Ngai-Man Cheung, Jun Fang, Ngoc-Bao Nguyen
A Class of Parameterized Loss Functions for Classification: Optimization Tradeoffs and Robustness Characteristics. Tyler Sypherd, Lalitha Sankar, Mario Diaz, Peter Kairouz, Gautam Dasarathy
Local Group Anomaly Detection for Multiple Time Series with Synchronization Errors. Yinxi Liu, Kai Yang, Shaoyu Dou
Understanding Thermodynamic Variational Inference. Rob Brekelmans, Aram Galstyan, Greg Ver Steeg
Phase Transitions for the Information Bottleneck. Tailin Wu, Ian Fischer
GAIT: A Geometric Approach to Information Theory. Jose Gallego Posada, Ankit Vani, Max Schwarzer, Simon Lacoste-Julien
Representation Learning via state aggregation: A perspective of control over communication channels. Aditya Mahajan, Jayakumar Subramanian
Autoregressive Models: What Are They Good For? Murtaza Dalal, Alexander Li, Rohan Taori
The Bethe approximation for structured matrices: an improved approximation for the profile maximum likelihood. Moses Charikar, Kirankumar Shiragur, Aaron Sidford
Fundamental Limits of Nonconvex Stochastic Optimization. Yossi Arjevani, Yair Carmon, John Duchi, Dylan Foster, Nathan Srebro, Blake Woodworth
Information-Theoretic Perspective of Federated Learning. Linara Adilova, Julia Rosenzweig, Michael Kamp
Exploring Unique Relevance for Mutual Information based Feature Selection. Shiyu Liu, Mehul Motani
Abstract
Information theory is deeply connected to two key tasks in machine learning: prediction and representation learning. Due to these connections, information theory has found wide applications in machine learning, such as proving generalization bounds, certifying fairness and privacy, optimizing information content of unsupervised/supervised representations, and proving limitations to prediction performance. Conversely, progress in machine learning has driven advancements in classical information theory problems such as compression and transmission.
This workshop aims to bring together researchers from different disciplines, identify common grounds, and spur discussion on how information theory can apply to and benefit from modern machine learning tools. Topics include, but are not limited to:
Controlling information quantities for performance guarantees, such as PAC-Bayes, interactive data analysis, information bottleneck, fairness, privacy. Information theoretic performance limitations of learning algorithms.
Information theory for representation learning and unsupervised learning, such as its applications to generative models, learning latent representations, and domain adaptation.
Methods to estimate information theoretic quantities for high dimensional observations, such as variational methods and sampling methods.
Quantification of usable / useful information, e.g. the information an algorithm can use for prediction.
Machine learning applied to information theory, such as designing better error-correcting codes, and compression optimized for human perception.
Key Dates
Submission starts: August 15, 2019
Extended abstract submission deadline: September 15, 2019 (23:59 AOE)
Acceptance notification: September 30, 2019 (23:59 AOE)
Camera ready submission: November 15, 2019
Workshop date: December 13, 2019
Speakers
Alessandro Achille (UCLA)
Alex Alemi (Google)
Varun Jog (University of Wisconsin-Madison)
Po-Ling Loh (University of Wisconsin-Madison)
Aaron van den Oord (DeepMind)
Ayfer Özgür (Stanford)
Stefano Soatto (UCLA)
Organizers
Camera Ready Instructions
Use the following modified NeurIPS style file, or include "Workshop on Information Theory and Machine Learning, " in front of the first page footnote.
https://drive.google.com/file/d/1--3XrNVcDzYXnK2hB8mPJLkJtn-PsDz7/view?usp=sharing
Use \usepackage[final]{itml2019} to include the style file.
Submit the camera ready version to CMT, with file name changed to [cmt paper submission id].pdf
Presentation Instructions
Posters should be no larger than 36W x 48H inches or 90 x 122 cm, and printed on light weight, not laminated paper. Tapes will be provided.
All accepted papers will receive a 1 minute spotlight presentation at the beginning of the poster session. Please upload a one page slide in PDF format to cmt as a supplementary file no later than Dec 8th. (Please address questions about spotlight slides to kechoi@cs.stanford.edu)