Sandeep Kumar - ELL 888 Advanced Machine Learning

ELL 888: Advanced Machine Learning (Foundations in High-Dimensional and Graph ML)

Instructors: Sandeep Kumar (SDK) and Jayadeva (JD)

3 credits (3-0-0)Pre-requisites: Linear Algebra, Probability, Introductory Machine Learning, and OptimizationSemester II: 2021-2022
Course Objective: This course will assume a background in the basics of linear algebra, machine learning, and optimization. The goal of this course is to train students with foundational concepts and skills in Machine Learning for high-dimensional, big data, non-Euclidean, irregular, and geometric data problems. The theory will go in conjunction with hands-on analysis of real-world applications, including ML, networks, learning, computer vision, bioinformatics, controls, etc. In the first part, mathematics for machine learning and learnability will be discussed. In the second part, machine learning techniques for high-dimensional and big data problems will be covered. The third part will introduce the rapidly evolving area of Geometric Machine Learning.

Research-Based Course:

The course is interdisciplinary in nature, it would welcome advanced undergraduate, master's, and Ph.D. students from various disciplines interested in the mathematical foundations and applications of machine learning for high-dimensional, big data, non-Euclidean, and geometric data.

Project: This course will have considerable weight on projects that might be of publishable quality. The students could pick topics from their domain, the project will aim to expose students to the state-of-art literature in the area and will be helpful for their research.

Guest Lecture Series:

Title: Stochastic Compositional Gradient Descent under Compositional constraints, Date 28/01/2022 and 4/02/2022, Speaker: Srujan Teja, Senior Ph.D. Scholar, IIT Kanpur.
Title: Elasticity of Neural Networks, Speaker: Prof. Anirbit Mukherjee, The University of Manchester, Date 11/02/2022 and 18/02/2022.
Title: Multi-Robot Trajectory Planning via Deep Reinforcement Learning, Speaker: Mohan Krishna, Senior Ph.D. Scholar, IIT Kanpur, Date 25/02/2022,
Title: Stochastic Gradients and its Variants, Speaker: Prof. Ketan Rajawat, IIT Kanpur, 25/03/2022 and 1/04/2022.
Title: Neural Network based End-to-End Query by Example Spoken Term Detection, Speaker Dr. Dhananjay Ram, Amazon Seattle US, Dates: 04/03/2022.
Title: Stochastic Non-Parametric Optimization, Speaker Dr. Amrit Singh Bedi, US Army Lab, Dates: Will be announced soon.

A Very Rough and Evolving Course Sketch:

Mathematical Preliminaries

Matrix Analysis: Eigen analysis, Rank Analysis, and Spectral Graph Theory
Probability: Exponential family of distribution, Sufficient Statistics

Overview of Statistical Estimation: MLE, MAP, and Bayes Principle

Machine Learning Theory

Foundational Aspects of Learning, PAC Learning, VC Dimension, Learnability
Structural and Empirical Risk
Minimizing Risk: Minimizing the VC dimension, Minimal Complexity Machines, Data Augmentation

High-Dimensional Machine Learning

Curse of Dimensionality
Sparsity
Parametric and non-Parametric High-Dimensional Regression and Classification
SVM and MCM
Kernel-Based Methods
Matrix Factorization and Low-Rank Approximation

Graph Machine Learning

Graph as Universal Data Structure

Laplacian matrix; Smoothness, Dirichlet Energy, Laplacian Beltrami operator
Graph Construction and Graph Learning from Data
Graph-based Semi-Supervised, Supervised, and Unsupervised ML algorithms
Graph Regularization and Label Propagation
Graph Embedding and Graph Representation Learning
Graph (Deep)Learning: Graph Neural Networks, Architecture, and Applications

Manifold Machine Learning

Basics of Manifolds: Stiefel, Grassmannian, and Riemannian Manifolds
Matrix Manifolds
Information Geometry
Optimization Methods for Manifolds
Machine Learning over Manifolds

Format

Scribing, Demo, and Slides (20%),
Project (40%) Individually or in pairs: outstanding performance in a project will be appropriately rewarded.
Mid-sem exam (15%)
End-sem exam (25%)

Readings:

John Hopcroft and Ravi Kannan, Foundations of Data Science, Hind Book Agency, TRIM Series, (2014).
Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. The Elements of Statistical Learning, Springer series in statistics, 2001.
Murphy, Kevin P. Machine Learning: A Probabilistic Perspective. MIT Press, 2021.
Deisenroth, Marc Peter, A. Aldo Faisal, and Cheng Soon Ong. Mathematics For Machine Learning. Cambridge University Press, 2020.
Koller, Daphne, and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques, MIT Press, 2009.
Boumal, Nicolas. "An Introduction to Optimization on Smooth Manifolds." Available online, Aug (2020).
Research Papers from JMLR, NeurIPS, ICML, IEEE TIT, IEEE TSP, IEEE TPAMI, etc.