Week 1 (8/29): Introduction, Continuous batching and KV cache management
Reading list:
Orca: A Distributed Serving System for Transformer-Based Generative Models (https://www.usenix.org/conference/osdi22/presentation/yu)
Efficient Memory Management for Large Language Model Serving with PagedAttention (https://dl.acm.org/doi/10.1145/3600006.3613165)
SGLang: Efficient Execution of Structured Language Model Programs (https://proceedings.neurips.cc/paper_files/paper/2024/file/724be4472168f31ba1c9ac630f15dec8-Paper-Conference.pdf)
Lab 1 out (due 9/2, 2025, 23:59 PDT)
CARC information: https://www.carc.usc.edu/user-guides/hpc-systems/discovery
Week 2 (9/5): Distributed Training
Reading list:
Efficient large-scale language model training on GPU clusters using megatron-LM (https://dl.acm.org/doi/10.1145/3458817.3476209)
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines (https://dl.acm.org/doi/pdf/10.1145/3458817.3476145)
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (https://dl-acm-org.libproxy2.usc.edu/doi/10.5555/3433701.3433727)
Extra reading
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (https://arxiv.org/pdf/1909.08053)
PipeDream: Generalized Pipeline Parallelism for DNN Training
Beyond Data and Model Parallelism for Deep Neural Networks
Lab 2 out (due 9/9, 2025, 23:59 PDT)
Week 3 (9/12): Attention
Discussion:
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (https://arxiv.org/abs/2307.08691)
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving (https://arxiv.org/pdf/2501.01005)
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (https://openreview.net/pdf?id=fPBACAbqSN)
Extra reading:
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Efficient Streaming Language Models with Attention Sinks
Week 4 (9/19): Efficient decoding + Ideas lab 1
Reading list:
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (https://www.usenix.org/conference/osdi24/presentation/agrawal)
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin)
Week 5 (9/26): Long-context generation
Reading list:
Ring Attention with Blockwise Transformers for Near-Infinite Context
InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
Week 6 (10/3): Mixture-of-experts
Reading list:
Accelerating Distributed MoE Training and Inference with Lina (https://www.usenix.org/conference/atc23/presentation/li-jiamin)
Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores (https://arxiv.org/abs/2503.10725)
MegaScale-Infer: Efficient Mixture-of-Experts Model Serving with Disaggregated Expert Parallelism [SIGCOMM'25]
SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling [NSDI'26]
Extra reading (background)
Switch transformer
OLMoE
Week 7 (10/10): Fall recess
Pre-proposal (1 page) due (10/8 11:59pm)
Week 8 (10/17): Pre-proposal presentation and team formation
10/18: Leaders decide on team members
10/20: Leaders finalize team members (must discuss with instructor to change team after this date)
Week 9 (10/24): Proposal feedback sessions w/ instructor (CREDIF Symposium 9-4pm, please attend! rsvp)
Full project proposal due (10/28 11:59pm)
Week 10 (10/31): Fault tolerance
Reading list:
ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development (https://www.usenix.org/conference/nsdi25/presentation/wan-borui)
ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation
Attack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training [NSDI'26] (https://arxiv.org/abs/2504.19232)
Week 11 (11/7): Early-exits
Reading list:
Improving DNN Inference Throughput Using Practical, Per-Input Compute Adaptation
Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving
Week 12 (11/14): Quantization and Sparsification
Reading list:
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration
DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization
T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge
SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs
Week 13 (11/21): Multi-modal, RLHF, and RAG
Reading list:
DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models [SIGCOMM'25]
Optimizing RLHF Training for Large Language Models with Stage Fusion
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
Week 14 (11/28): Thanksgiving holiday
Week 15 (12/5): Heterogeneous cluster
Reading list:
Hi-Speed DNN Training with Espresso: Unleashing the Full Potential of Gradient Compression with Near-Optimal Usage Strategies (https://dl.acm.org/doi/10.1145/3552326.3567505)
SkyServe: Serving AI Models across Regions and Clouds with Spot Instances (https://dl.acm.org/doi/10.1145/3689031.3717459)
Final's week (12/10-17): Project presentation and demo