Algorithms for Large-scale Distributed Machine Learning and Optimization

Topic

1 - Intro and Logistics

2 - SGD and its Variants in Machine Learning

3 - SGD Convergence Analysis

4 - Variance-reduced SGD, Distributed Synchronous SGD

5 - Asynchronous SGD, Hogwild

 6 - Local-update SGD

7 - Adacomm, Elastic Averaging, Overlap SGD

8 - Quantized and Sparsified Distributed SGD

9 - Decentralized SGD

10 - Federated Learning Intro

11 - Data Heterogeneity in FL

12 - Computational Heterogeneity in FL

13 - Client Selection and Partial Participation

14 - Personalized Federated Learning

15 - Multi-task Learning

16 - Federated Min-max Optimization

17 - Fairness and Participation Incentives

18 - Differential Privacy in Dist. Optimization

19 - Secure Aggregation in Distributed Learning

20 - Robustness to Adversaries

21 - Federated Learning in the LLM Era