CS 203: Software Tools & Techniques for AI
IIT Gandhinagar
Sem II - 2024-25
Instructor: Mayank Singh (email: singh.mayank@iitgn.ac.in)
Class Schedule: K1, Monday 15:30 - 16:50
Location: AB 10/201
Lab Schedule: J2,M2, Friday 14:00 - 15:20 15:30 - 16:50
Location: AB 7/109
Communication Google group: cs203-2024.pvtgroup@iitgn.ac.in
Google Classroom: CS 203 Google Classroom join this classroom for class related communication.
Teaching Assistants:
Himanshu Beniwal (himanshubeniwal@iitgn.ac.in)
Eshwar Dhande (eshwar.dhande@iitgn.ac.in)
Aamod Thakur (aamod.thakur@iitgn.ac.in)
Sailesh Panda (24210086@iitgn.ac.in)
Prerequisite (Optional)
Computing ES 112
Course Contents
1. Data collection and labeling
a. collecting data: tools for logging/instrumenting sources (e.g. website) to collect usage data;
b. validating data
c. labeling data: tools such as label studio to annotate datasets, methods such as Cohen Kappa inter-annotator agreement, snowball sampling, active learning for cost-effective data annotation,
d. data augmentation
2. Model and code reproducibility and versioning:
a. architecture and hyperparameter search: tools such as weights and biases for architecture and hyperparameter search and various AutoML tools such as AutoGluon
b. model checkpointing: tools such as MLFlow, Tensorboard/Weights, and Biases for model/weights checkpointing
c. data versioning;
3. Self-contained environments: tools such as Docker for managing released packages; cloud platforms such as GCP/AWS/Azure;
4. Run-time profiling and optimization (e.g., memory footprint, measuring GPU utilization and optimizing it).
5. Continuous Integration/Co: A/B tests; Distribution, Covariate shift— data version management and retraining to account for shifts; Redeployment
6. Deployment on constrained devices: tools such as TF-lite, topics like quantization, etc.
Lecture Slides and Lab Material (Slides and Colab Notebooks)
Week 1 (Jan 6-10): Course Introduction and Data Collection (Lead TA:Eshwar)
Lecture:
Course overview and objectives are provided by a quick walkthrough of the course website.
Introduction to data collection: logging/instrumenting tools, ethical considerations.
Tools: Google Analytics, OpenTelemetry, and Python logging.
Lab:
Hands-on with instrumenting a website or app to collect usage data.
Setting up simple logging systems to store structured logs. [Assignment 1]
Week 2 (Jan 13-17): Data Validation (Lead TA: Sailesh)
Lecture:
Importance of data validation: quality, completeness, and consistency. [Slides (taken verbatim from Introduction to Data-Centric AI course at MIT)]
Tools: Great Expectations, Pandora.
Lab:
Validate datasets using Great Expectations.
Create validation pipelines for common formats (CSV, JSON). [Assignment 2]
Week 3 (Jan 20-24): Data Labeling and Inter-Annotator Agreement (Lead TA:Aamod)
Lecture:
Overview of data labeling tools (e.g., Label Studio).
Methods for inter-annotator agreement (e.g., Cohen’s Kappa). [Slides]
Lab:
Annotating datasets using Label Studio.
Implementing and calculating inter-annotator agreement. [Assignment 3]
Week 4 (Jan 27-31): Active Learning and Cost-Effective Labeling (Lead TA:Eshwar)
Lecture:
Active learning for reducing annotation costs.
Strategies: uncertainty sampling, query-by-committee. [Slides, (taken verbatim from Digging into Data course at University of Maryland)] [Slides]
Lab:
Implement an active learning loop for a labeling task.
Evaluate the cost-effectiveness of active learning vs. random sampling. [Assignment 4]
Week 5 (Feb 3-7): Data Augmentation (Lead TA:Aamod)
Lecture:
Data augmentation techniques (e.g., image, text, audio).
Tools: Albumentations, Augly. [Slides]
Lab:
Apply data augmentation to a dataset. [Assignment 5]
Compare model performance with and without augmentation.
Week 6 (Feb 10-14): Reproducibility and Versioning (Lead TA:Eshwar)
Lecture:
Importance of reproducibility in AI workflows. [Slides]
Tools for tracking experiments: Weights & Biases, MLflow. [Colab Notebook]
Lab:
Set up experiment tracking for a simple machine-learning model.
Version control of datasets using DVC (Data Version Control).
Week 7 (Feb 17-21): Architecture and Hyperparameter Search (Lead TA: Himanshu)
Lecture:
Automated architecture and hyperparameter optimization.[Slides]
Tools: Weights & Biases sweeps, AutoGluon.
Lab:
Perform hyperparameter optimization using AutoGluon.
Compare manual tuning vs. automated search.
Week 8 (Feb 24-28): Model Checkpointing (Lead TA: Himanshu)
Lecture:
Managing model checkpoints: saving and restoring training states. [slides]
Tools: Checkpointing in Pytorch. [Colab]
Lab:
Implement checkpointing in a PyTorch training loop. [Assignment 7]
Week 9 (Mar 3-7): Mid-Semester Exams
Week 10 (Mar 10-14): Self-Contained Environments (Lead TA: Aamod)
Lecture:
Importance of self-contained environments for reproducibility. [slides]
Tools: Docker, virtual environments. [Code to create flask Image]
Lab:
Create a Docker container for a simple AI application.
Explore deployment options on GCP/AWS.
Week 11 (Mar 17-21): Profiling and Optimization (Lead TA: Eshwar)
Lecture:
Profiling runtime: GPU utilization, memory usage. [Slides]
Optimization strategies: mixed precision, model pruning.
Lab:
Profile a model using tools like PyTorch Profiler.
Apply optimizations to improve performance.
Week 12 (Mar 24-28): Continuous Integration/Continuous Deployment (CI/CD) (Lead TA: Sailesh)
Lecture:
CI/CD in AI pipelines: retraining for distribution shifts.
Tools: GitHub Actions, Jenkins. [Slides taken from Steven Grunwell's talk, Understanding MLOps]
Lab:
Set up a simple CI pipeline to train a model.
Implement a workflow for automatic model retraining and deployment.
Week 13 (Mar 31- Apr 4): A/B Testing and Covariate Shift (Lead TA: Sailesh)
Lecture:
A/B testing strategies in AI models.
Handling distribution and covariate shifts. [Slides, taken verbatim from CS109a: Introduction to Data Science]
Lab:
Implement A/B testing on model predictions.
Use tools like Alibi Detect to handle covariate shifts.
Week 14 (Apr 7-11): Deployment on Constrained Devices (Lead TA: Himanshu)
Lecture:
Deploying models on resource constraint devices.
Techniques: quantization. [Google Colab]
Lab:
Convert and deploy a model using TensorFlow Lite. [Assignment 11]
Optimize for size and speed on a constrained device.
Week 15 (Apr 14-18): Course Wrap-Up and Discussions
Lecture:
Summary of key concepts covered in the course.
Discussion of trends and future directions in AI tools and techniques.
Lab: Brainstorming Lab sessions
Week 16 (Apr 21 onwards): Examinations and Evaluations
Grading Policy & Schedule
12 Lab Assignments (60%)
Two Written quizzes (10%)
One Mid/End-semester Exam (20%)
10 Surprise Attendances (10%)
Books
Murphy, K.P. Probabilistic machine learning: Advanced topics. MIT press. 2022.
Murphy, K.P. Probabilistic machine learning: an introduction. MIT press. 2022.
Prince, S.J. Understanding Deep Learning MIT PRESS. 2023.
Gift, Noah, and Alfredo Deza. Practical MLOps. O'Reilly Media, Inc. 2021.
Ron Kohavi, Diane Tang, Ya Xu. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. 2020.
Géron, Aurélien. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow. O'Reilly Media, Inc. . 2019. [2nd Edition]
Reference course
CS 329 course from Stanford University https://stanford-cs329s.github.io/syllabus.html