ELL 780: Mathematical Foundations for Computer Technology
ELL 457: Special Topics in Cognitive & Intelligent System
Instructors: Sandeep Kumar (SDK)
3 credits (3-0-0)Pre-requisites: Elementary Calculus and Matrix Theory.Semester I: 2021-2022Evaluation: Scribing (15%), Quiz (25%), Minor Exam (25%), and Major Exam (35%).
Course Objective: Fundamental mathematical topics like real analysis, linear algebra, statistics, probability, and optimization, are essential for data science, machine learning, artificial intelligence, and allied areas. These topics are generally taught in disparate courses, making it difficult for students to hone their skills in these areas. The aim of this course is to bring the foundational topics, mainly real analysis, matrix theory, and convex optimization to the fore and deliver the information in a self-contained manner. The course will start with a comprehensive introduction to the fundamental mathematical topics, which will be followed by discussions on models and algorithms. Numerous applications from the data science domain will be covered. This course will be particularly helpful to students from an engineering background, who seek to build their expertise in the aforementioned area.
Course Sketch:
Analysis: Introduction to Metric Spaces, Vector Spaces, Normed Linear Spaces, and Inner Product Spaces. Dual Norm. Continuity, Smoothness, Lipscticz property, Differentiability, Sequence, and Convergence.
Matrix Theory: System of Linear Equations, Linear Transformation, Eigenvalues, and Eigenvectors, Singular Value Decomposition (SVD), Pseudo Inverse. Positive (Semi)Definite Matrices and Associated Properties, Quadratic Forms and Least Squares, Low-Rank Approximation, Graph Matrices, and its Spectral Properties.
Optimization Basics: Convex Set and Convex Function. Optimization Formulation, Convex Optimization, Different types of Optimization Problem Formulation (LP, QP, QCQP, SOCP). Optimality Condition, Lagrangian Duality, KKT Conditions, Introduction to First-Order Methods.
Textbooks:
- W.Rudin, Principles of Mathematical Analysis Vol. 3. New York: McGraw-hill, 1976.
- Gilbert Strang, Linear Algebra and Learning from Data, Wellesley-Cambridge Press, (2019.)
- Stephen P. Boyd, and Lieven Vandenberghe, Convex Optimization, Cambridge university press, (2004).
Additional Readings:
- S. Chandra, Jayadeva and Aparna Mehra, Numerical Optimizations with Applications, Narosa, (2014).
- G. H. Golub and C. F. Van-Loan, Matrix Computations, John Hopkins Studies in Computational Science, (2013).
- E.Kreyszig, Introductory Functional Analysis with Applications, Wiley(1978).
- Charu C. Aggarwal, Linear Algebra and Optimization for Machine Learning, Springer, (2020).
- John Hopcroft and Ravi Kannan, Foundations of Data Science, Hind Book Agency, TRIM Series, (2014).
Tentative Lecture Plan
Real Analysis
Sets, Functions, and Countability
Real numbers, Vector Space, and Subspace
Norms, Dual Norms, and Operator Norm
Metrics, Metric Space, and Inner Product Space
Convergence, Limit, Continuity, and Differentiability
Matrix Analysis
System of Linear Equations, Matrix Multiplication, and Matrix Rank
Four Fundamental Spaces
Least Squares and Inverse of a Matrix
Orthogonality, Gram-Schmidt Process, and Projections
Matrix Factorization: EVD, SVD, LU, and QR
Positive Definite Matrices
Convex Optimization
Optimization: Motivation and Formulation
Convex Sets
Convex Functions
Convex Optimization Problem Formulations
Optimality Condition
Lagrange Duality and KKT Conditions
Convex Optimization Formulations: LP, QP, QCQP, SDP, SOCP
Solving Optimization Problems with Examples
First-Order Methods
Projection Based Methods