Distributed Machine Learning
Distributed Machine Learning
Instructor: Dr. Ferdous Pervej
Email: ferdous.pervej@usu.edu
E-mail Turnaround Time: 24 Hours
Website: https://sites.google.com/view/mfpervej/ece-5930-6930
Telephone: 435-797-9549 (office)
Office hours: TBA (at least 1 hour dedicated office hour and open door policy)
Lecture time: TBA
Lecture location: EL - TBA
None. Course materials will span largely from different sources (e.g., papers, book chapters). However, the following materials will be helpful:
Amir Beck, “First-Order Methods in Optimization,” Society for Industrial and Applied Mathematics, 2017, ISBN978-1-61197-498-0.
L´eon Bottou, Curtis E. Frank, and Nocedal Jorge, “Optimization methods for large-scale machine learning,” SIAM review 60.2 (2018) (Free online copy: https://epubs.siam.org/doi/epdf/10.1137/16M1080173)
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, “Foundations of Machine Learning,” second edition, MIT press, 2018. (Free online copy: https://cs.nyu.edu/ mohri/mlbook/)
MAE 5370 or
CS 4320 or
CS 5640 or
CS 5665 or
MATH 2270/2275 and MATH 2210 and MATH 5710
This course offers theory of machine learning (ML) and optimizations. Students will learn different topics of first-order optimization using various traditional and ML algorithms, followed by large-scale distributed ML algorithms, privacy-preserving federated learning (FL) algorithms, and communication- and computation-efficient distributed learning.
Introduction (≈ 3 lectures)
General topics (e.g., convex and non-convex function, smoothness, strong convexity, empirical risk function, gradients, subgradients, hessians, Jacobian, generalization errors, level sets, etc.)
First-order methods (≈ 8 lectures)
Gradient descent, mini-batch SGD, and convergence analysis
Momentum and variance reduction methods
Subgradient methods
Mirror descent
Proximal gradient methods
Distributed optimization and machine learning (≈ 16 lectures)
Introduction to distributed machine learning
Distributed SGD
Introduction to communication-efficient learning
Federated averaging (FedAvg)
Data and computation heterogeneity
FedProx
FedNova
Scaffold
FedProto
Communication-efficient learning - advanced topics
Client selections and partial device participation
Gradient compression/quantization
Computation-efficient learning
Pruning
Split learning
Information leakage and advanced privacy-preserving methods (differential privacy, secure aggregation)
This course provides foundations to ML, first-order optimizations, and distributed ML. The key objectives are
To study various first-order optimization techniques
Study various centralized and distributed ML algorithms
Study large-scale ML in distributed networks
Learn communication- and computation-efficient distributed learning
Study advanced privacy-preserving techniques to prevent gradient leakage
Additional and Substantive Learning Objectives for 6000-level Students
In accordance with Utah System of Higher Education (USHE), the graduate students who are enrolled in the ECE 6930 section will have additional and substantive learning objectives as listed below.
Learn joint optimization techniques for running distributed ML in practical networks (e.g., LTE, 5G, WiFi-7)
Emulate distributed ML algorithms in practical wireless networks (e.g., LTE, 5G, WiFi-7)
There will be additional question(s) added to at least two homework assignments and additional requirements in at least one
project to achieve the two additional substantive learning objectives listed above.
The grade will be distributed according to the following weights:
Item | Weight
Quiz (Closed book/lecture) | 5 %
Homework | 20 %
Mini-projects | 50 %
Semester-long projet | 25 %
Late submissions (homework and projects) will be accepted up to 1 week after the due date with a 30% penalty unless prior arrangements are made.
Students who wish to challenge or request that their instructor/TA review their grades on any homework, project, or midterm exam must request within seven days from when the grade was posted. For the final exam, students must request within 24 hours of when the grade was posted.
In cooperation with the Disability Resource Center (DRC), reasonable accommodation will be provided for qualified students with disabilities. Please reach out to the instructor prior to the first class (or during the first week of class) to make arrangements. Alternate format print materials (large print, audio, diskette or Braille) will be available through the Disability Resource Center.
USU welcomes students with disabilities. If you have, or suspect you may have, a physical, mental health, or learning disability that may require accommodations in this course, please contact the Disability Resource Center (DRC) as early in the semester as possible (University Inn #101, 435-797-2444, drc@usu.edu). All disability related accommodations must be approved by the DRC. Once approved, the DRC will coordinate with faculty to provide accommodations.
Utah State University supports the principle of freedom of expression for both faculty and students. The University respects the rights of faculty to teach and students to learn. Maintenance of these rights requires classroom conditions that do not impede the learning process. Disruptive classroom behavior will not be tolerated. An individual engaging in such behavior may be subject to disciplinary action. Read Student Code Article V Section V-3 for more information.
Each student has the right and duty to pursue his or her academic experience free of dishonesty. To enhance the learning environment at Utah State University and to develop student academic integrity, each student agrees to the following Honor Pledge:
“I pledge, on my honor, to conduct myself with the foremost level of academic integrity.”
A student who lives by the Honor Pledge is a student who does more than not cheat, falsify, or plagiarize. A student who lives by the Honor Pledge:
Espouses academic integrity as an underlying and essential principle of the Utah State University community;
Understands that each act of academic dishonesty devalues every degree that is awarded by this institution; and
Is a welcomed and valued member of Utah State University.
The instructor of this course will take appropriate actions in response to Academic Dishonesty, as defined the University’s Student Code. Acts of academic dishonesty include but are not limited to:
Cheating: using, attempting to use, or providing others with any unauthorized assistance in taking quizzes, tests, examinations, or in any other academic exercise or activity. Unauthorized assistance includes:
Working in a group when the instructor has designated that the quiz, test, examination, or any other academic exercise or activity be done “individually;”
Depending on the aid of sources beyond those authorized by the instructor in writing papers, preparing reports, solving problems, or carrying out other assignments;
Substituting for another student, or permitting another student to substitute for oneself, in taking an examination or preparing academic work;
Acquiring tests or other academic material belonging to a faculty member, staff member, or another student without express permission;
Continuing to write after time has been called on a quiz, test, examination, or any other academic exercise or activity;
Submitting substantially the same work for credit in more than one class, except with prior approval of the instructor; or engaging in any form of research fraud.
Falsification: altering or fabricating any information or citation in an academic exercise or activity.
Plagiarism: representing, by paraphrase or direct quotation, the published or unpublished work of another person as one’s own in any academic exercise or activity without full and clear acknowledgment. It also includes using materials prepared by another person or by an agency engaged in the sale of term papers or other academic materials.
For additional information go to: ARTICLE VI. University Regulations Regarding Academic Integrity.