Instructor: Avishek Ghosh, Assistant Professor, Dept. of CSE, IIT Bombay
Contact: Room KR-218, Kresit Building; Email: avishek@cse.iitb.ac.in
Timing: Monday/Thursday 5:30-7:00 pm
Classroom: CC-103 [This is in the new Computer Science Building]
TA: (1) Garima Jain (jgarima203@gmail.com), (2) Sravani Gunnu (gunnu.sravi@gmail.com)
About the course: The content is roughly divided into 2 parts:
Classical Theory of Learning:
1. Empirical Process Theory: Empirical Risk Minimization, Concentration of Measure, Hoeffdings, Macdiarmid`s, Azuma, Bernstein lemma, Generalization.
2. Uniform Law of large numbers: Glivenko Cantelli, Rademacher Averages, Covering, Packing, Metric Entropy, VC dimension,Chaining, Dudley Entropy Integral.
3. M estimation: Rates of convergence for M estimators, Consistency, Localization, Lipschitz regression.
4. Sparse linear regression: Gaussian sequence model, convex penalized regression, Hard and Soft thresholding estimator, LASSO, rates of convergence for LASSO.
Modern Theory of Learning
1. Statistical Learning theory for Deep Learning, Interpolation, Overparameterization
Grading: HWs (10%), Quizz (10%), Mid-sem (35%), End-sem (35%), and Class participation (10%)
References:
1. High Dimensional Statistics, A non-asymptotic approach-Martin J Wainwright, Cambridge University Press, 2022.
2. Deep learning: a statistical viewpoint; Bartlett, Montenari and Rakhlin (https://arxiv.org/pdf/2103.09177)
3. The Elements of Statistical Learning Theory-T. Hastie, R. Tibshirani, J. Friedman, Springer, 2009.
4. Empirical Processes in M estimation-Sara Van de Geer, Cambridge University Press, 2000.
5. All of Statistics-Larry A. Wasserman, Springer, 2004.
6. Concentration Inequalities-S. Boucheron, P. Massart, G. Lugosi, Oxford Press, 2013.
7. High Dimensional Probability-R. Vershynin, Cambridge University Press, 2018
Apart from these references, we will follow the lecture notes of Prof. Peter Bartlett (UC Berkeley), Prof. Aditya Guntuboyina (UC Berkeley), Prof. Arya Mazumdar (UC San Diego) on similar courses.
Discussions:
We will use Moodle for discussions and communication. Please enroll if you haven't yet.
General Guidelines for Homeworks:
Homeworks should be submitted in class. Students are encouraged to discuss among themselves while solving the HW problems. However, they are required to write the solution on their own. Near-identical submissions will not be awarded any score.