Organization

• Instructor: Hui Guan

• Course number: COMPSCI 690AB

• Class meetings: LGRC A104A TT 2:30-3:45 PM

• Office hours information: Please check Canvas

• Communications: Canvas and CampusWire 

• Syllabus: You can read the syllabus here. 

Class overview

This course is designed to provide a comprehensive understanding of computer systems architecture that supports deep learning workloads. It assumes students have prior knowledge on computer systems, algorithms, and Python/C/C++ programming background. In the course, we will study the full-stack system design to support deep learning, covering topics from the high-level programming frameworks to low-level kernel implementations. We will also introduce cutting-edge research on efficient and scalable deep learning model training, inference, and serving. 

Course schedule

Below is the schedule for this semester and will be updated throughput the semester. 

02/01 Intro and Logistic [Slides]

• Reading Assignment: None

• Suggested Reading: 

  • How to read a paper

02/06 DNN Background I [Slides][Recording]

• Topics Covered: DNN overview, MLP, CNN

• Reading Assignment: None

• Suggested Reading: 

  • this deep learning book is great to get a comprehensive view of the deep learning field. 

02/08 DNN Background II [Slides][Recording]

• Topics Covered: CNN, LeNet, RNN, Transformer Encoder 

• Reading Assignment: None

• Suggested Reading: 

  • This deep learning book is great to get a comprehensive view of the deep learning field. 

02/13 DNN Background III [Slides][Recording] 

• Topics Covered: Transformer Encoder, Decoder

• Reading Assignment: None

• Suggested Reading: 

  • Attention is all you need 

02/15 System Perspectives [Slides][Recording]

• Topics Covered: Latency, Throughput, Model Complexity Metrics, MLSys conference 

• Reading Assignment: Write a summary of the following paper 

  • MLSys: The New Frontier of Machine Learning Systems

• Suggested Reading: 

  • A Berkeley View of Systems Challenges for AI

02/20 Computation Frameworks [Slides][Recording] 

• Topics Covered: Requirements of Computation Frameworks, Abstractions, Dataflow graph, Automatic Differentiation 

• Reading Assignment: None

• Suggested Reading: 

  • Automatic Differentiation in Machine Learning: a Survey

• Homework 1 released after class. Due March 7th EoD!

  • Open with Google Colab

02/22 NO CLASS 

• Monday class schedule will be followed

02/27 Computation Framework II [Slides][Recording]

• Topics Covered: Dataflow graph optimization, Static vs dynamic dataflow graph, Tensorflow, Tensorflow Eager, PyTorch 

• Reading Assignment: 

  • TensorFlow: A System for Large-Scale Machine Learning

• Suggested Reading: 

  • Automatic Differentiation in Machine Learning: a Survey  

  • TensorFlow Eager: A multi-stage, Python-embedded DSL for machine learning 

  • PyTorch: An Imperative Style, High-Performance Deep Learning Library

02/29 Kernel Implementation I [Slides][Recording] 

• Topics Covered: DNN Operators, CPU and Matrix Multiplication, SIMD, GPU architectures, SIMT

• Reading Assignment: NONE

• Suggested Reading: 

  • GPU MM Background User' Guide

  • Storage and the Memory Hierarchy 

03/05 Project Proposal Day 

• Introduce your project in class in 2-3 minutes to get feedback and suggestion from others.

• Submit your 1-page project proposal on Canvas in one week

• You can find the project guidelines here. 

03/07 Kernel Implementation II [Slides][Recording] 

• Topics Covered: GPU programming, kernel, thread blocks, GPU and matrix multiplication, tiling, FlashAttention 

• Reading Assignment: 

  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS'22 

• Suggested Reading: 

  • FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

• HW1 DUE EoD

03/12 Inference Overview [Slides][Recording] 

• Topics Covered: Overview of inference systems, performance metrics, optimization techniques 

• Reading Assignment: 

  • MLPerf Inference Benchmark

• Suggested Reading: 

  • Deep Learning Inference in Facebook Data Centers

  • Deep learning inference service at microsoft 

• Project Proposal DUE EoD

03/14 Pruning [Slides][Recording]

• Topics Covered: MLPerf, Pruning Overview 

• Suggested Reading: 

  • SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

  • A Simple and Effective Pruning Approach for Large Language Models, ICLR'24

  • Wootz: a Compiler-based Framework for Fast CNN Pruning via Composability

• Homework 2 released this week. Due Thursday, April 4th EoD!

  • Open with Google Colab

03/19 NO CLASS -- Spring Break 

03/21 NO CLASS -- Spring Break 

03/26 Pruning (Cont.) [Slides][Recording] 

• Topics covered: Pruning granularity, pruning criterion, FlashLLM discussion 

• Reading Assignment: 

  • Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity, VLDB'24

• Suggested Reading: 

  • SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

  • A Simple and Effective Pruning Approach for Large Language Models, ICLR'24

  • Wootz: a Compiler-based Framework for Fast CNN Pruning via Composability

03/28 Quantization [Slides][Recording] 

• Topics covered: Data types, linear quantization, quantization-aware training, post-training quantization 

• Reading Assignment: 

  • The case for 4-bit precision: k-bit Inference Scaling Laws, ICML'23

• Suggested Reading: 

  • GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, ICLR'23 

  • SqueezeLLM: Dense-and-Sparse Quantization

  • Post-Training 4-bit Quantization on Embedding Tables

04/02 Model Compilation [Slides][Recording]

• Topics covered: compiler, machine learning compilation, TVM 

• Reading Assignment: 

  • TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

• Suggested Reading: 

  • Machine Learning Compilation

  • AI Compilers Demystified

  • A friendly introduction to machine learning compilers and optimizers

  • The Deep Learning Compiler: A Comprehensive Survey

04/04 Data Parallel Training [Slides][Recording] 

• Topics covered: distributed training overview, data parallel training, ZeRO

• Reading Assignment: None 

• Suggested Reading 

  • Zero: Memory optimizations toward training trillion parameter models, SC'20

04/09 Model Parallel Training [Slides][Recording] 

• Topics covered: tensor parallelism, pipeline parallelism, comparisons 

• Reading Assignment: None 

• Suggested Reading 

  • Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism  

  • Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning, OSDI'22 

04/11 Guest Lecture: On-device inference [Slides][Recording] 

• Reading Assignment: 

  • Re-thinking computation offload for efficient inference on IoT devices with duty-cycled radios

• Suggested Reading: 

  • Efficient IoT Inference via Context-Awareness

  • Low-latency Speculative Inference On Distributed Multi-modal Data Streams

04/16 Efficient Finetuning [Slides][Recording] 

• Reading Assignment: None 

• Suggested Reading: 

  • LoRA: Low-Rank Adaptation of Large Language Models, ICLR'21

  • QLoRA: Efficient Finetuning of Quantized LLMs

04/18 Guest Lecture: Model Serving [Slides][Recording] 

• Reading Assignment 

  • Proteus: A High-Throughput Inference-Serving System with Accuracy Scaling, ASPLOS'24

• Suggested Reading 

  • Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP'23

  • INFaaS: Automated Model-less Inference Serving, ATC'21

  • Nexus: A GPU Cluster Engine for Accelerating DNN-Based Video Analysis, SOSP'19

04/23 - 05/09 Project Presentation 

• Present your course project in class in 10 minutes + 2 min Q/A.

• Submit your project report May 17th. 

Note: You can also find the tentative planned schedule of an entire semester in this spreadsheet. The spreadsheet is tentative and subject to change!

Accommodation Statement

The University of Massachusetts Amherst is committed to providing an equal educational opportunity for all students. If you have a documented physical, psychological, or learning disability on file with Disability Services (DS), you may be eligible for reasonable academic accommodations to help you succeed in this course. If you have a documented disability that requires an accommodation, please notify me within the first two weeks of the semester so that we may make appropriate arrangements.  

Academic Honesty Statement

Since the integrity of the academic enterprise of any institution of higher education requires honesty in scholarship and research, academic honesty is required of all students at the University of Massachusetts Amherst. Academic dishonesty is prohibited in all programs of the University. Academic dishonesty includes but is not limited to: cheating, fabrication, plagiarism, and facilitating dishonesty. Appropriate sanctions may be imposed on any student who has committed an act of academic dishonesty. Instructors should take reasonable steps to address academic misconduct. Any person who has reason to believe that a student has committed academic dishonesty should bring such information to the attention of the appropriate course instructor as soon as possible. Instances of academic dishonesty not related to a specific course should be brought to the attention of the appropriate department Head or Chair. Since students are expected to be familiar with this policy and the commonly accepted standards of academic integrity, ignorance of such standards is not normally sufficient evidence of lack of intent (http://www.umass.edu/dean_students/codeofconduct/acadhonesty/ ).