Embedded Files
>> Module Contributor
>> Module Contributor
Jia (Kevin) Liu
Associate Professor
Department of Electrical and Computer Engineering
The Ohio State University
>> Module Description
>> Module Description
This module will introduce algorithm design and convergence analysis in nonconvex optimization theory as well as their applications in solving modern machine learning and data science problems. The goal of this module is to prepare graduate students with a solid theoretical and mathematical foundation at the intersection of optimization and machine learning so that they will be able to use optimization to solve advanced machine learning problems and/or conduct advanced research in the related fields. This module will take the traditional linear, nonlinear, and convex optimization taught in operation research or related engineering fields (e.g., ECE, CSE) as a prerequisite, and focus on topics in nonconvex optimization that are of special interest in the machine learning community.
>> Module Details
>> Module Details
Module
Topic
Notes
2. First-Order Methods for Nonconvex Optimization
2-1. Math Background Review
2-2. Convexity
2-3. Gradient Descent
2-4. Stochastic Gradient Descent (General Expectation Minimization, Finite-Sum Minimization)
2-5. Variance-Reduced Methods (SAG, SVRG, SAGA, SPIDER, PAGE)
2-6. Adaptive Methods (AdaGrad, RMSProp, Adam)
3. Federated and Decentralized Learning
3-1. Federated Learning (Distributed Learning, FedAvg)
3-2. Decentralized Learning (Decentralized SGD, Gradient Tracking)
4. Zeroth-Order Methods for Nonconvex Optimization
4-1. ZO Methods with Random Directions of Gradient Estimation
4-2. Variance-Reduced Zeroth-Order Methods
5. First-Order Nonconvex Optimization with Special Geometric Structure
5-1. The PL Condition and NTK
5-2. NTK and Weak-Quasi-Convexity
Page updated
Google Sites
Report abuse