This course covers fundamentals in compilation techniques used in the domain of machine learning. The topics include tensor programming languages, frameworks, compilers, tensor intermediate representations, code generation for GPU and specialized accelerators. We will also cover tensor program optimizations, automatic differentiation, techniques for approximating neural networks such as pruning and quantization, and compilers for probabilistic ML models. We will discuss principles behind the modern toolchains for machine learning, such as JAX, MLIR, Pytorch 2.0, TVM, XLA.
Course objectives:
Upon completion of the course students will be able to:
Understand the programming models and intermediate representations of commonly used tensor algebra compilers.
Understand and develop compiler analysis and transformation passes for domain specific tensor algebra compilers.
Understand and develop machine learning techniques to make automatic optimization decisions for compiler optimization problems.
Understand and develop approximate compilation techniques for ML workloads.
Communication:
Most of class communication and the latest news will be on the course's Campuswire. Email Yifeng if you need access to the site.
Lectures:
Tuesday and Thursday
11am-12:15pm
Siebel Center 0220 (Urbana)
200 Wecker Dr Classroom 4.C (Chicago)
Quick Links:
Campuswire
Zoom (Illinois login)
Instructors:
Charith Mendis (charithm@illinois.edu)
Sasa Misailovic (misailo@illinois.edu)
Teaching Assistant:
Yifeng Ding (yifeng6@illinois.edu)
Office Hours:
Charith: by appointment
Sasa: by appointment
Yifeng: Wednesday 3pm-5pm
(Siebel basement and Zoom)
Course Format: The class will meet two times a week. Most of the class sessions will be instructor driven lectures. Students can participate in-person or remotely over Zoom. We will also include guest lectures and/or paper discussions to cover evolving topics as content evolves for this area. The students will complete three structured class projects, two quizzes, and one open-ended research project.
The coursework will be self-contained, but we expect the students to have basic background in machine learning (e.g., at the level of CS 441) and compilers (e.g., at the level of CS 421 or CS 426). We expect that the students are familiar with Python and C++.
This is the first offering of this course. We try to be transparent and predictable, but we maintain the right to make any changes deemed to be in the best interest of the students, and will communicate all such changes as promptly as possible.
Course Requirements & Assessment:
Take-home Quizzes (each 10% of the grade): Mid-semester and end-of-semester quizzes.
Project 1 (10% of the grade): Manually optimizing matrix multiplication operator on CPU, GPU and on an accelerator (Individual)
Project 2 (20% of the grade): Compiler transformation and ML cost model (Individual).
Project 3 (10% of the grade): Differentiable and probabilistic programming (Individual).
Project 4 (40% of the grade): Open-ended research project (Done in groups of two or three).
Grading will be on the absolute scale, based on the obtained number of points. We will not use the curve for determining the grade; we are happy to give A to the entire class if everyone completes their assignments. We will give ample opportunities for extra credit, up to 10% of the total points, on many of the assignment listed above. We will give A+ to a small number of students who have shown exemplary performance throughout the class. However, this will be limited to less than 10% of the class.
Letter Grades:
Letter grades will be assigned based on the following thresholds: 93 (A); 90 (A-); 87 (B+); 83 (B); 80 (B-); 77 (C+); 70 (C); 67 (C-); 60 (D+); 53 (D); over 50 (D-); 50 and under (F). We may lower the thresholds if warranted by the grade distribution (but never raise the thresholds).
The detailed schedule will evolve as the course progresses. The slides and the lecture recordings will be available on Campuswire.
Date
Topic
Presenter
1/21
Introduction
Charith/Sasa
1/23
Background on Deep Neural Networks and Tensors
Sasa
1/28
Tensors and Tensor Programming Languages
Charith
1/30
Introduction to Tensor Compilers used in Machine Learning
Charith
2/4
Kernel-level Code Generation
Charith
2/6
Hardware-aware Kernel Programming (part 1)
Charith
2/11
Hardware-aware Kernel Programming (part 2)
Charith
2/13
Graph-level Optimizations (part 1)
Charith
2/18
Guest Lecture "Computing Systems for Artificial Intelligence"
2/20
Graph-level Optimizations (part 2)
Charith
2/25
Graph-level Optimizations (part 3)
MP1 Deadline February 25
Charith
2/27
Hardware-aware code-generation (part 1)
MP2 Released
Charith
3/4
Hardware-aware code-generation (part 2)
Charith
3/6
Quiz 1 (take-home)
---
3/11
ML Compiler Testing
Sasa
3/13
Model-specific Optimizations
MP2 Deadline March 14
Charith
Spring Break (no classes)
---
3/25
Automatic Differentiation 1
Sasa
3/27
Automatic Differentiation 2
MP2 Deadline March 28
Sasa
4/1
Automatic Differentiation 3
Sasa
4/3
Approximations of Neural Networks: Motivation and Pruning
MP3 Released
Sasa
4/8
Approximations of Neural Networks: Quantization
Sasa
4/10
Approximations of Neural Networks for the Edge
Sasa
4/15
Probabilistic Programming 1
Sasa
4/17
Probabilistic Programming 2
MP3 Deadline April 18
Sasa
4/22
Compiling and Autotuning ML for the Edge and TinyML
Sasa
4/24
ML for Compilers
Charith
4/29
Quiz 2 (take home)
---
5/1
TBD
Charith/Sasa
5/6
Guest Lecture
Charith/Sasa
Time for Project Submission and Lateness:
The deadline for each project submission is 11:59pm Champaign Standard Time.
Students may use up to 72 hours of extra time to submit their work without penalty. The students can distribute this time across any of three projects (no extension for MP4).
The granularity of extra time is 1hr and you do not need to ask the instructors.
After the grace extra time, the penalty is 2% of the number of points for that assignment for each late hour.
The students do not need to inform the teaching staff in advance if they are using this extra time. The teaching staff will keep track of the submission times based on the timestamps on the server.
There are three circumstances for which we'll give "free" extensions:
Serious personal problems. These will be dealt with individually. They are typically either serious illness to you (doctor's verification required) or a major family emergency.
Major paper deadlines. These will also be dealt with individually. Your professor/research advisor should send us a short note in advance.
Extended lab downtime. This is extremely rare. A whole day of downtime in the lab might qualify. Minor downtime does not qualify. "Flakiness all weekend" does not qualify. Accidental loss of files does not qualify. If you have chosen to work on some other machine other than the designated class machines, then you do so at your own risk. To protect yourself, keep recent backups on the class machines so that if your machine breaks, you can recover.
We will allow using tools like ChatGPT, GPT-4, Copilot and similar to assist you while programming in a limited capacity. Today's computing world evolves and these tools will enter the development cycle sooner or later. However, a fair warning: those tools are just tools and can both help and hurt your work. You will be fully responsible for the correctness of the code. Currently, it is unlikely that these tools can generate fully (or even mostly) accurate code. The necessity for you to do systematic testing will increase, and debugging time and cost may exceed those if you were writing the code by hand from scratch.
Note 1: You will have to keep tracks of the logs of the conversations with tools and submit them with each machine project for which you used these tools. The prompts can be in PDF or text format. As prompt engineering is emerging as a discipline, you will be among the first to try to systematically use it in your project.
Note 2: The most effective way to use the LLMs will be to ask small questions and combine and modify the responses to fit your goal. On the other hand, we disallow the prompts that are not about technical content, but are leaning toward cheating (e.g., "Show me the code solution for optimized GPU implementation of Matmul on Github"). Such uses are not permitted in this class and will fall under the cheating policy.
Note 3: We reserve the right to disallow using these tools for specific projects if we identify that their use will not serve the educational goals. In such cases, we will make this constraint clear at the time we publish the problem description. As the tools constantly evolve, it is difficult to predict their behavior in several weeks or months from now.
Using LLM-based development tools will not be an excuse for identical project codes and other major similarities that would be qualified as plagiarism. Even with the tools' help there will be a need for significant editing of the projects that will very likely result in a unique project code. The teaching staff will also do their tests on the extent some of these tools generate usable code. In the case there is a doubt of cheating (see the policy below), your detailed logs of prompts may be helpful.
Cheating:
You are NOT to copy solutions from ANY source (including, but not limited to, code available on the Web, code written in previous semesters, or code obtained from people other than your partner). While the pressures of many classes, homework, work and/or extracurricular activities can be great, this is never an excuse for copying solutions from others. "Helping" somebody by allowing them to copy part of your code is not doing them a favor either, but indicates your approval and active participation in such activities. Refer to the Campus Code regarding academic integrity. In particular, giving someone else code is just as much cheating as copying someone else's code. Cheating will result in a reduced grade, or an "F" grade for the course. If you are aware of any breach of academic integrity, it is your responsibility to report it to the instructor.
Discussing your problems with other students, however, is encouraged and even highly recommended. This means asking someone not in your team to give you advice, asking them for suggestions when you face an obstacle, or just discussing any aspect of the material. What you may not do is to copy or in any way use code written by anyone else or dictated by anyone else not on your team. As responsible adults, you can draw the distinction between cheating and honest behavior: exercise your judgement and try to preserve the highest level of professional integrity in everything you do.
Bottom line: You may discuss difficulties with others and get occasional help with debugging but you or your team must write your code entirely by yourself. In your report, list the other students with whom you discussed the solution.
Participate actively in class: join the discussions and ask questions when you don't understand something.
Think about the project as a research problem to tackle and solve, not an MP with canned procedures and goals.
When writing code in large frameworks, it is easy to get intimidated and bogged down because of the quantity of code that exists. The critical lesson to learn is to determine what parts of the code you have to read and learn, and what you don't. You should try to learn what you actually need to get the project done, but also a little bit more. Read some of the functions you use often. Look for existing code fragments to use as patterns (but don't follow them rigidly: figure out how to write your code best). Then buckle down and write your own code, and don't read much more or you will get bogged down.
Read assigned papers before coming to class.
Start the project assignments early, and don't leave all your studying to the days before each exam. Working madly to beat a deadline is a terrible way to learn!
Mental Health:
Diminished mental health, including significant stress, mood changes, excessive worry, substance/alcohol abuse, or problems with eating and/or sleeping can interfere with optimal academic performance, social development, and emotional wellbeing. The University of Illinois offers a variety of confidential services including individual and group counseling, crisis intervention, psychiatric services, and specialized screenings at no additional cost. If you or someone you know experiences any of the above mental health concerns, it is strongly encouraged to contact or visit any of the University’s resources provided below. Getting help is a smart and courageous thing to do -- for yourself and for those who care about you.
Counseling Center: 217-333-3704, 610 East John Street Champaign, IL 61820
McKinley Health Center:217-333-2700, 1109 South Lincoln Avenue, Urbana, Illinois 61801
Sexual Misconduct Reporting Obligation:
The University of Illinois is committed to combating sexual misconduct. Faculty and staff members are required to report any instances of sexual misconduct to the University’s Title IX Office. In turn, an individual with the Title IX Office will provide information about rights and options, including accommodations, support services, the campus disciplinary process, and law enforcement options.
A list of the designated University employees who, as counselors, confidential advisors, and medical professionals, do not have this reporting responsibility and can maintain confidentiality, can be found here: http://wecare.illinois.edu/resources/students/#confidential
Other information about resources and reporting is available here: http://wecare.illinois.edu.
Academic Integrity:
The University of Illinois at Urbana-Champaign Student Code should also be considered as a part of this syllabus. Students should pay particular attention to Article 1, Part 4: Academic Integrity. Read the Code at the following URL: http://studentcode.illinois.edu/.
Academic dishonesty may result in a failing grade. Every student is expected to review and abide by the Academic Integrity Policy: https://studentcode.illinois.edu/article1/part4/1-401/. Ignorance is not an excuse for any academic dishonesty. It is your responsibility to read this policy to avoid any misunderstanding. Do not hesitate to ask the instructor(s) if you are ever in doubt about what constitutes plagiarism, cheating, or any other breach of academic integrity.
Religious Observances:
Illinois law requires the University to reasonably accommodate its students' religious beliefs, observances, and practices in regard to admissions, class attendance, and the scheduling of examinations and work requirements. You should examine this syllabus at the beginning of the semester for potential conflicts between course deadlines and any of your religious observances. If a conflict exists, you should notify your instructor of the conflict and follow the procedure at https://odos.illinois.edu/community-of-care/resources/students/religious-observances/ to request appropriate accommodations. This should be done in the first two weeks of classes.
Disability-Related Accommodations:
To obtain disability-related academic adjustments and/or auxiliary aids, students with disabilities must contact the course instructor and the Disability Resources and Educational Services (DRES) as soon as possible. To contact DRES, you may visit 1207 S. Oak St., Champaign, call 333-4603, e-mail disability@illinois.edu or go to https://www.disability.illinois.edu. If you are concerned you have a disability-related condition that is impacting your academic progress, there are academic screening appointments available that can help diagnosis a previously undiagnosed disability. You may access these by visiting the DRES website and selecting “Request an Academic Screening” at the bottom of the page.
Emergencies:
Emergencies can happen anywhere and at any time, so it’s important that we take a minute to prepare for a situation in which our safety could depend on our ability to react quickly. Take a moment to learn the different ways to leave this building. If there’s ever a fire alarm or something like that, you’ll know how to get out and you’ll be able to help others get out. Next, figure out the best place to go in case of severe weather – we’ll need to go to a low-level in the middle of the building, away from windows. And finally, if there’s ever someone trying to hurt us, our best option is to run out of the building. If we cannot do that safely, we’ll want to hide somewhere we can’t be seen, and we’ll have to lock or barricade the door if possible and be as quiet as we can. We will not leave that safe area until we get an Illini-Alert confirming that it’s safe to do so. If we can’t run or hide, we’ll fight back with whatever we can get our hands on. If you want to better prepare yourself for any of these situations, visit police.illinois.edu/safe. Remember you can sign up for emergency text messages at emergency.illinois.edu.
Family Educational Rights and Privacy Act (FERPA):
Any student who has suppressed their directory information pursuant to Family Educational Rights and Privacy Act (FERPA) should self-identify to the instructor to ensure protection of the privacy of their attendance in this course. See https://registrar.illinois.edu/academic-records/ferpa/ for more information on FERPA.
Writer's Workshop:
The Writers Workshop’s mission is to support the University of Illinois community by providing free writing assistance for students, faculty, and staff from all disciplines and at all stages of the writing process. Whether you’re working on a course paper, senior capstone, personal statement, group writing project, thesis or dissertation, or manuscript for publication, their consultants are available to help. Visit them at any stage of the writing process: brainstorming, organizing ideas, polishing final drafts, citing sources, and more. The Workshop offers online appointments, in-person appointments, and evening drop-in hours. The Workshop also sponsors writing groups and provides presentations about academic writing skills. Find more information at writersworkshop.illinois.edu.