Teaching

Dr Sparsh taught "Hardware Architectures for Deep Learning" course. Here are its course-contents. 

Background Topics: 

* Floating point and fixed point numbers

* Approximate computing and storage

* Cache tiling (blocking)

* GPU architecture, Tensor Cores, NVIDIA's FP4/FP8 formats, GraceHopper Superchip, NVLink and NVSwitch, 

* FPGA architecture

* Matrix multiplication using systolic array

* VLIW architecture

* Qualcomm's Hexagon DSP architecture

* Huffman encoding. 

* Formats for storing sparse matrices

* Superscalar, out-of-order and Pipelining

* Multiprocessing, multithreading and SIMD

* Study of LineShine: Number 1 Supercomputer in World as of 2026

Background Topics on Deep-learning:

* Compute and memory techniques for optimizing CNNs

* Convolutional strategies: Direct, FFT-based, Winograd-based and Matrix-multiplication based.

* Review of “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding”

* Size-aware and Hardware-aware pruning of DNNs. Review of "Scalpel: Customizing DNN Pruning to the Underlying Hardware Parallelism."

* Prefill and Decode: Two key phases of LLMs

Deep Learning on Various Processors/Accelerators

* Deep Learning on CPUs

* Deep learning on FPGAs

* Case study of Microsoft's Brainwave

* Deep learning on Embedded System (especially NVIDIA's Jetson Platform)

* Deep learning on Edge Devices (smartphones). Review of “Machine Learning at Facebook: Understanding Inference at the Edge”.

* Case Study of Google's Tensor Processing Unit (version 1 to 3)

* Study of Meta's MTIA accelerator (version 1 and 2; maybe version 3)

* Study of Cerebras Wafer Scale Engine 

Deep Learning in Various Use cases

* Hardware/system-challenges in autonomous driving. Review of "The Architectural Implications of Autonomous Driving: Constraints and Acceleration". 

* Mixture-of-experts and their scheduling on CPU-GPU heterogeneous computing systems

* Agentic LLMs and their scheduling on SoCs

 Additional Topics If Time Permits: 

* Neural branch predictor. Review of "Using Branch Predictors to Predict Brain Activity in Brain-Machine Implants" 

* Distributed training of DNNs. Review of  "Optimizing Network Performance for Distributed DNN Training on GPU  Clusters: ImageNet/AlexNet Training in 1.5 Minutes"

* Roofline Model