CS 203: Software Tools & Techniques for AI

IIT Gandhinagar

Sem II - 2024-25

Instructor: Mayank Singh (email: singh.mayank@iitgn.ac.in)

Class Schedule: K1, Monday 15:30 - 16:50

Location: AB 10/201

Lab Schedule: J2,M2, Friday 14:00 - 15:20 15:30 - 16:50

Location: AB 7/109 

Communication Google group: cs203-2024.pvtgroup@iitgn.ac.in

Google Classroom: CS 203 Google Classroom   join this classroom for class related communication.

Teaching Assistants:

1. Himanshu Beniwal (himanshubeniwal@iitgn.ac.in)

2. Eshwar Dhande (eshwar.dhande@iitgn.ac.in)

3. Aamod Thakur (aamod.thakur@iitgn.ac.in)

4. Sailesh Panda (24210086@iitgn.ac.in)

Prerequisite (Optional)

• Computing ES 112

Course Contents

1. Data collection and labeling 

a. collecting data: tools for logging/instrumenting sources (e.g. website) to collect usage data; 

b. validating data 

c. labeling data: tools such as label studio to annotate datasets, methods such as Cohen Kappa inter-annotator agreement, snowball sampling, active learning for cost-effective data annotation, 

d. data augmentation

2. Model and code reproducibility and versioning: 

a. architecture and hyperparameter search: tools such as weights and biases for architecture and hyperparameter search and various AutoML tools such as AutoGluon 

b. model checkpointing: tools such as MLFlow, Tensorboard/Weights, and Biases for model/weights checkpointing 

c. data versioning;

3. Self-contained environments: tools such as Docker for managing released packages; cloud platforms such as GCP/AWS/Azure;

4. Run-time profiling and optimization (e.g., memory footprint, measuring GPU utilization and optimizing it).

5. Continuous Integration/Co: A/B tests; Distribution, Covariate shift— data version management and retraining to account for shifts; Redeployment

6. Deployment on constrained devices: tools such as TF-lite, topics like quantization, etc.

Lecture Slides and Lab Material (Slides and Colab Notebooks)

Week 1 (Jan 6-10): Course Introduction and Data Collection  (Lead TA:Eshwar)

              Lecture: 

1. Course overview and objectives are provided by a quick walkthrough of the course website.

2. Introduction to data collection: logging/instrumenting tools, ethical considerations. 

3. Tools: Google Analytics, OpenTelemetry, and Python logging.

             Lab: 

1. Hands-on with instrumenting a website or app to collect usage data.

2. Setting up simple logging systems to store structured logs. [Assignment 1]

Week 2 (Jan 13-17): Data Validation  (Lead TA: Sailesh)

               Lecture: 

1. Importance of data validation: quality, completeness, and consistency. [Slides (taken verbatim from Introduction to Data-Centric AI course at MIT)]

2. Tools: Great Expectations, Pandora.

               Lab:

1. Validate datasets using Great Expectations.

2. Create validation pipelines for common formats (CSV, JSON). [Assignment 2]

Week 3 (Jan 20-24): Data Labeling and Inter-Annotator Agreement  (Lead TA:Aamod)

              Lecture: 

1. Overview of data labeling tools (e.g., Label Studio).

2. Methods for inter-annotator agreement (e.g., Cohen’s Kappa). [Slides]

              Lab:

1. Annotating datasets using Label Studio.

2. Implementing and calculating inter-annotator agreement. [Assignment 3]

Week 4 (Jan 27-31): Active Learning and Cost-Effective Labeling  (Lead TA:Eshwar)

              Lecture: 

1. Active learning for reducing annotation costs.

2. Strategies: uncertainty sampling, query-by-committee. [Slides, (taken verbatim from Digging into Data course at University of Maryland)] [Slides]

              Lab:

1. Implement an active learning loop for a labeling task.

2. Evaluate the cost-effectiveness of active learning vs. random sampling. [Assignment 4]

Week 5 (Feb 3-7): Data Augmentation  (Lead TA:Aamod)

              Lecture: 

1. Data augmentation techniques (e.g., image, text, audio).

2. Tools: Albumentations, Augly. [Slides]

              Lab:

1. Apply data augmentation to a dataset. [Assignment 5]

2. Compare model performance with and without augmentation.

Week 6 (Feb 10-14): Reproducibility and Versioning  (Lead TA:Eshwar)

              Lecture: 

1. Importance of reproducibility in AI workflows. [Slides]

2. Tools for tracking experiments: Weights & Biases, MLflow. [Colab Notebook]

              Lab:

1. Set up experiment tracking for a simple machine-learning model.

2. Version control of datasets using DVC (Data Version Control).

Week 7 (Feb 17-21): Architecture and Hyperparameter Search  (Lead TA: Himanshu)

              Lecture: 

1. Automated architecture and hyperparameter optimization.[Slides] 

2. Tools: Weights & Biases sweeps, AutoGluon.

              Lab:

1. Perform hyperparameter optimization using AutoGluon. 

2. Compare manual tuning vs. automated search.

Week 8 (Feb 24-28): Model Checkpointing  (Lead TA: Himanshu)

              Lecture: 

1. Managing model checkpoints: saving and restoring training states. [slides]

2. Tools: Checkpointing in Pytorch. [Colab]

              Lab:

1. Implement checkpointing in a PyTorch training loop. [Assignment 7]

Week 9 (Mar 3-7): Mid-Semester Exams

Week 10 (Mar 10-14): Self-Contained Environments  (Lead TA: Aamod)

Lecture: 

1. Importance of self-contained environments for reproducibility. [slides]

2. Tools: Docker, virtual environments. [Code to create flask Image]

              Lab:

1. Create a Docker container for a simple AI application.

2. Explore deployment options on GCP/AWS.

Week 11 (Mar 17-21): Profiling and Optimization  (Lead TA: Eshwar)

Lecture: 

1. Profiling runtime: GPU utilization, memory usage. [Slides]

2. Optimization strategies: mixed precision, model pruning.

              Lab:

1. Profile a model using tools like PyTorch Profiler.

2. Apply optimizations to improve performance.

Week 12 (Mar 24-28): Continuous Integration/Continuous Deployment (CI/CD)  (Lead TA: Sailesh)

    Lecture: 

1. CI/CD in AI pipelines: retraining for distribution shifts.

2. Tools: GitHub Actions, Jenkins. [Slides taken from Steven Grunwell's talk, Understanding MLOps]

              Lab:

1. Set up a simple CI pipeline to train a model.

2. Implement a workflow for automatic model retraining and deployment.

Week 13 (Mar 31- Apr 4):  A/B Testing and Covariate Shift (Lead TA: Sailesh)

    Lecture: 

1. A/B testing strategies in AI models.

2. Handling distribution and covariate shifts. [Slides, taken verbatim from CS109a: Introduction to Data Science]

              Lab:

1. Implement A/B testing on model predictions.

2. Use tools like Alibi Detect to handle covariate shifts.

Week 14 (Apr 7-11): Deployment on Constrained Devices  (Lead TA: Himanshu)

              Lecture: 

1. Deploying models on resource constraint devices.

2. Techniques: quantization. [Google Colab]

              Lab:

1. Convert and deploy a model using TensorFlow Lite. [Assignment 11]

2. Optimize for size and speed on a constrained device.

Week 15 (Apr 14-18): Course Wrap-Up and Discussions

              Lecture: 

1. Summary of key concepts covered in the course.

2. Discussion of trends and future directions in AI tools and techniques.

              Lab: Brainstorming Lab sessions

Week 16 (Apr 21 onwards): Examinations and Evaluations

Grading Policy & Schedule

• 12 Lab Assignments (60%)

• Two Written quizzes (10%)

• One Mid/End-semester Exam (20%) 

• 10 Surprise Attendances (10%)

Books

• Murphy, K.P. Probabilistic machine learning: Advanced topics. MIT press. 2022.

• Murphy, K.P. Probabilistic machine learning: an introduction. MIT press. 2022.

• Prince, S.J. Understanding Deep Learning MIT PRESS. 2023.

• Gift, Noah, and Alfredo Deza. Practical MLOps. O'Reilly Media, Inc.  2021.

• Ron Kohavi, Diane Tang, Ya Xu. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. 2020.

• Géron, Aurélien. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow. O'Reilly Media, Inc.  . 2019. [2nd Edition]

Reference course

CS 329 course from Stanford University https://stanford-cs329s.github.io/syllabus.html