ELL 706: Optimization for EE

Instructors:  Prof.  Sandeep Kumar (SK) 

Course Objective:  Optimization tools have become an indispensable part of current research frontiers, it has been applied to a wide variety of problems in engineering applications, especially in signal processing, communications, machine learning, control, and networks.  This course covers the development and analysis of optimization algorithms.

This course will have considerable weight on term papers, that might be of publishable quality. The students could pick topics from their domain, and term paper will expose students to the state-of-art literature in the area and will be helpful for their research. The course is interdisciplinary in nature, it would welcome advanced undergraduate, masters and Ph.D. students from various departments.

 Course Sketch:

• Convex set, convex functions, convex optimization formulations, Optimality conditions, Lagrangian Duality, KKT Conditions.

• Applications of optimization in concrete real-world problems

•  First-order optimization methods: gradient methods, subgradient methods, proximal methods, and block-coordinate descent methods.

•  Regularization methods for structural optimization: sparsity, low rank, graphical, smoothness, shrinkage, margin, manifold learning.

•  Large-scale optimization techniques: Distributed Optimization Frameworks; Alternate Minimization and Majorization-Minimization methods

• Stochastic optimization: Stochastic gradient descent and variance reduced methods.

 References:

• Stephen P. Boyd, and Lieven Vandenberghe, Convex Optimization,  Cambridge university press, (2004).

•  Amir Beck, First-Order Methods in Optimization, 2011.David Luenberger, Optimization by Vector Space Methods

•  David Luenberger, Optimization by Vector Space Methods

•  Survit Sra, Sebastian Nowozin, and Stephen J. Wright, eds. Optimization for machine learning. MIT Press, 2012.

•  Prateek Jain and Purushottam Kar. Non-convex optimization for machine learning." Foundations and Trends in Machine Learning, (2017).

•  Selected research papers.