Generative Modeling

• Auto-Encoding Variational Bayes (VAE). Many variations:

  • beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

  • Neural Discrete Representation Learning (VQ-VAE)

  • Learning Disentangled Joint Continuous and Discrete Representations  (Joint-VAE)

• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

• Large Scale Adversarial Representation Learning

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Vision Transformers)

• Denoising Diffusion Probabilistic Models

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model

• LoRA: Low-Rank Adaptation of Large Language Models

Graph Embeddings

"Textbooks"

• Graph Representation Learning, Will Hamilton (2020)

• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges, Bronstein et al. (2021)

Spectral Methods / Random Walks

• (Blog) An Overview of k-way Spectral Clustering, Yeh (2021)

• Partitioning Well-Clustered Graphs: Spectral Clustering Works!, Peng et al. (2015)

• node2vec: Scalable Feature Learning for Networks, Grover and Leskovec (2016)

Graph Neural Networks

• (Blog) A Gentle Introduction to Graph Neural Networks, Sanchez-Lengeling et al. (2021)

• (Blog) Understanding Convolutions on Graphs, Daigavane et al. (2021)

• Message-Passing: Neural Message Passing for Quantum Chemistry, Gilmer et al. (2017)

• GCN: Semi-Supervised Classification with Graph Convolutional Networks, Kipf and Welling (2017)

• GIN: How Powerful are Graph Neural Networks?, Xu et al. (2018)

• GAT: Graph Attention Networks, Veličković et al. (2018)

• GATv2: How Attentive are Graph Attention Networks?, Brody et al. (2022)

• Strategies for Pre-training Graph Neural Networks

Self-Supervised Learning

• Unsupervised Representation Learning by Predicting Image Rotations

• A Simple Framework for Contrastive Learning of Visual Representations 

• What Makes for Good Views for Contrastive Learning? 

• Representation Learning with Contrastive Predictive Coding

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

• Shuffle and Learn: Unsupervised Learning using Temporal Order Verification

• Unsupervised Learning of Object Landmarks through Conditional Image Generation

• Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Symbolic & Neurosymbolic Representations

• Neurosymbolic Programming (review paper)

• HOUDINI: Lifelong Learning as Program Synthesis 

• Learning Neurosymbolic Generative Models via Program Synthesis

• Learning Differentiable Programs with Admissible Neural Heuristics

• Synthesizing Programs for Images using Reinforced Adversarial Learning 

• DeepCoder: Learning to Write Programs

• DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning 

• Unsupervised Learning of Neurosymbolic Encoders

Representation Learning with Applications

Vision and Language

• (language) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

• (language) ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

• (vision & language) Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

• (vision & language) Learning Transferable Visual Models From Natural Language Supervision

• (vision & language) Zero-Shot Text-to-Image Generation

• (vision) Benchmarking Representation Learning for Natural World Image Collections

• (vision) Transfusion: Understanding Transfer Learning for Medical Imaging

Behavior Analysis

• Task Programming: Learning Data Efficient Behavior Representations

• Composing graphical models with neural networks for structured representations and fast inference 

• VAE-SNE: a deep generative model for simultaneous dimensionality reduction and clustering

• Interpreting Expert Annotation Differences in Animal Behavior

• Self-Supervised Keypoint Discovery in Behavioral Videos 

Chemistry

• MSA Transformer

• ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

• Molecular Contrastive Learning of Representations via Graph Neural Networks

• 3D Molecular Representations Based on the Wave Transform for Convolutional Neural Networks

• OrbNet: Deep Learning for Quantum Chemistry Using Symmetry-Adapted Atomic-Orbital Features

Dynamics Modeling & Physics

• Discovering governing equations from data by sparse identification of nonlinear dynamical systems

• Neural Point Process for Learning Spatiotemporal Event Dynamics

• Machine Learning for Fluid Mechanics

• Deep learning for universal linear embeddings of nonlinear dynamics

• Fourier neural operator for parametric partial differential equations

• Deep learning in fluid dynamics

• FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators

• Neural General Circulation Models for Weather and Climate