Lectures

Lecture 1: Introduction to Data Science & EEPS 1960D

Lecture Slides:

• What is Machine Learning? (Brown only)

• Course Logistics (Brown only)

Readings

• [3 pages] Blei and Smyth (2017), Science and Data Science. PNAS. [link] (Open access)

Lecture 2: Introduction to Machine Learning

Lecture Slides

• The Machine Learning Workflow (Brown only)

Readings

• [optional] ISL: Chapter 2 (ISL: Introduction to Statistical Learning with Applications in R -- see Resources page)

• [optional] Electoral Precedent (xkcd)

Lecture 3: Mathematical Foundations

Lecture Slides

• Mathematical Foundations of Machine Learning [outline] [annotated]* (Brown only) *updated after Lecture 4

Pre-lecture activities

1. [10 minutes] Watch this video ("Vectors, what even are they?") to review the concept of a vector. All students should watch this, even if you believe you are already familiar with the concept of a vector.

2. Review additional linear algebra topics, as necessary. When deciding which topics to review, ask yourself whether you are familiar with both the "Physics student" view and the "Computer Science student" view of the concept and have the ability to translate between them. [Credit: linear algebra review videos were created by Grant Sanderson]

   • [10 minutes] Basis vectors, Linear combination, Span, Linear (in)dependence

   • [11 minutes] Matrices, Matrix-vector multiplication, Linear transformations

   • [10 minutes] Matrix multiplication

   • [4 minutes] Rank, Null space (The full video also includes linear systems and matrix inverse -- these are useful concepts to know but not required.)

   • [4 minutes] Non-square matrix, transformations between dimensions

   • [2 minutes] Dot product (inner product) (Watch to 2:10 for basic definition of the dot/inner product. The rest of the video (optional, but recommended) covers projections, vectors as matrices/transformations, and duality.)

   • [13 minutes] Change of basis (Critical concept: a vector (data) can be represented using different choices of basis and it is possible to translate the representation of a vector in one basis to its representation in another.)

   • [10 minutes] Eigenvectors, Eigenvalues, Eigenbasis (Focus on 0:00-6:00 and 13:04-16:28, and don't worry too much about determinants or computing eigenvectors/values).

   • If you're not into videos, you are welcome to review the following: Linear Algebra Review and Reference [pdf] by Zico Kolter and Chuong Do, for Stanford University's Machine Learning Course (CS229)

3. Review calculus topics, as necessary.

   • [6 minutes] Multivariable functions

   • [5 minutes] Transformations I; [4 minutes] Transformations II

   • [5 minutes] Gradient (definition); [6 minutes] Direction of steepest ascent

   • [8 minutes] Multivariable maxima and minima

4. Review probability and statistics topics, as necessary.

   • Seeing Theory interactive visualizations of concepts in probability and statistics

     • [15 minutes] Ch. 1 Expectation, Variance, Ch. 2 Conditional Probability & Ch. 3 Random Variables -- the other chapters are useful but not required.

Additional resources

• [optional] Math for Machine Learning [pdf] (Hal Daumé III, University of Maryland)

• [optional] Machine Learning Math Essentials [pdf] (Jeff Howbert, University of Washington)

Lecture 4: Clustering Part 1

Lecture Slides

• Intro to Unsupervised Learning and Cluster Analysis (Brown Only)

• Hierarchical Clustering and K-means Clustering (Brown Only)

• Lecture 3 (Mathematical Foundations) link above updated to include remaining statistics slides covered in Lecture 4

Resources

• Visualizing K-Means Clustering interactive demo (by Naftali Harris)

• Do These AI-created Fake People Look Real to You? (New York Times)

• Soares et al. (2018). The use of hierarchical clustering for the design of optimized monitoring networks, Atmos. Chem. Phys.

• [optional] ISL: Section 10.3 (Clustering) (ISL: Introduction to Statistical Learning)

• [optional] ESL: Section 14.3 (Clustering) (ESL: Elements of Statistical Learning)

Lecture 5: Clustering Part 2

Lecture Slides

• Project Brainstorm: When not to use Machine Learning (Brown Only)

• K-means Clustering (part 2) (Brown Only)

Resources

• scikit-learn Clustering: Covers K-means, Hierarchical clustering, DBSCAN and other clustering algorithms available in sklearn.cluster module.

Lecture 6: Dimensionality Reduction, Part 1

Lectures Slides [Recordings available under Media Library on Canvas]

• DBSCAN Clustering

• Intro to Dimensionality Reduction

• Principal Component Analysis (PCA)

• Nonnegative Matrix Factorization (NMF)

Resources

• Visualizing DBSCAN Clustering interactive demo (by Naftali Harris)

  • DBSCAN paper: Ester et al. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise, KDD.

• ISL Section 10.2 (PCA), ESL Section 14.5 (PCA)

  • The SVD and Low-Rank Matrix Approximations [pdf] (by Tim Roughgarden & Gregory Valiant, Stanford University, for CS 168)

  • Example (PCA for high-dimensional data): Novembre et al. (2008). Genes Mirror geography within Europe, Nature.

  • Empirical Orthogonal Function (EOF) Analysis (by Dennis Shea, NCAR)

• ESL Section 14.6 (NMF)

  • NMF paper: Lee & Seung (1999). Learning parts of objects by non-negative matrix factorization, Nature.

Lecture 7: Dimensionality Reduction (Part 2) and Case study

Lectures Slides [Recordings available under Media Library on Canvas]

• Nonlinear Dimensionality Reduction

  • Autoencoder

  • Self-Organizing Maps

  • t-SNE

• Case study: Unsupervised Learning in Marine Ecology

Resources

• Case study paper: Sonnewald et al. (2020). Elucidating ecological complexity: Unsupervised learning determines global marine eco-provinces, Science Advances.

• How to Use t-SNE Effectively interactive paper (by Wattenberg et al)

  • t-SNE paper: van der Maaten & Hinton (2008). Visualizing Data using t-SNE, Journal of Machine Learning Research (JMLR).

• Introduction to autoencoders (by Jeremy Jordan)

• Self Organizing Maps: Fundamentals [pdf] and Properties and Applications [pdf] (from Neural Computation course notes by John Bullinaria)

Lecture 8: Introduction to Supervised Learning

Lecture Slides

• Introduction to Supervised Learning

Resources

• ISL Chapter 2: Statistical Learning

Lecture 9: KNN and Linear Regression

Lecture Slides

• KNN and Linear Regression

Resources

• ISL Sections 2.2.3 (The Classification Setting), 4.6.4 (K-Nearest Neighbors)

• ISL Chapter 3: Linear Regression

• A Complete Guide to KNN with Applications in Python and R (by Kevin Zakka)

Lecture 10: Cross-validation and Regularization

Lecture Slides

• Cross Validation and Regularization (Ridge & LASSO)

Resources

• ISL Section 5.1: Cross-validation

• Cross-validation example: KNN for Iris Dataset [opens Colab Notebook]

• ISL Section 6.2: Shrinkage Methods

• Chapter 2: The Lasso for Linear Models, in Statistical Learning with Sparsity: The Lasso and Generalizations [pdf] (by Hastie, Tibshirani and Wainwright)

Lecture 11: Regularization (Part 2)

Lecture Slides

• Regularization (Part 2)

Resources

• Understanding Maximum Likelihood an interactive visualization (by Kristoffer Magnusson)

• ESL Section 8.2: The Bootstrap and Maximum Likelihood Methods

Lecture 12: Logistic Regression & Support Vector Machines (SVMs)

Lecture Slides

• Linear Classifiers and SVMs (Part 1)

Resources

• ISL Section 4.3: Logistic Regression

• ISL Chapter 9: Support Vector Machines

• Support Vector Machines interactive demo (by Jonas Greitemann)

Lecture 13: SVMs and Classification Metrics

Lecture Slides

• Support Vector Machines (SVMs)

Resources

• ISL Chapter 9: Support Vector Machines

• Support Vector Machines interactive demo (by Jonas Greitemann)

• Lecture Notes (CS 229 @ Stanford): Kernel Methods & Support Vector Machines [pdf] (by Andrew Ng)

• ROC curve demonstration (by Alex Rogozhinkov)

Lecture 14: Artificial Neural Networks

Lectures

• Artificial Neural Networks

Resources

• Tinker with a Neural Network in your Browser interactive demo (by Google)

• Recommended Videos (by Grant Sanderson) [note that these are sponsored videos containing promotions]

  • But what is a Neural Network?

  • Gradient descent, how neural networks learn

  • What is backpropagation really doing?

Lecture 15: Decision Trees and Ensemble Methods, Part 1

Lecture Slides

• Decision Trees, Part 1: CART, Random Forests

Resources

• ISL Chapter 8: Tree-Based Methods

• ISL 5.2: The Bootstrap

• A Visual Introduction to Machine Learning (a scrollable visualization of a decision tree)

• Seeing Theory: The Bootstrap (visualizing bootstrapping)

• Ensemble Learning: Bagging & Boosting

Lecture 16: Decision Trees and Ensemble Methods, Part 2

Lecture Slides

• Decision Trees, Part 2: MARS, boosting, air quality example

• Missing Data & Imputation

Resources

• Gradient Boosting explained (series of demos on trees and boosting)

• Gradient Boosting Interactive Playground (interactive demo by same author as above)

• Competent Jurors and Weak Learners (The Condorcet Jury Theorem)

• Example paper: PM2.5 Prediction Based on Random Forest, XGBoost, and Deep Learning Using Multisource Remote Sensing Data

• ISL Section 9.6: Missing Data

• Imputing Missing Values in scikit-learn

Lecture 17: Machine Learning Review, Tricks & Tips

Lecture Slides

• Machine Learning Review: A Few Useful Things to Know About Machine Learning

• Imbalanced Data

Resources

• Reading: Domingos (2012). A Few Useful Things to Know About Machine Learning [pdf], Communications of the ACM.

Lecture 18: Gaussian Process Regression

Lecture Slides

• Gaussian Processes

Resources

• A Visual Exploration of Gaussian Processes (by Görtler et al)

• Understanding Gaussian Processes & Fitting a Gaussian Process Kernel (by Peter Roelants)

• Gaussian Processes, not quite for dummies (by Yuge Shi)

• Reference text: Gaussian Processes for Machine Learning [pdfs] by Rasmussen and Williams

Lecture 19: Kaggle Competition and GPR examples

Lecture Slides

• Kaggle Competition and GPR

Resources

• Domingo et al. (2020). Using Ice Cores and Gaussian Process Emulation to Recover Changes in the Greenland Ice Sheet During the Last Interglacial. JGR Earth Surface. [link]

• How to (almost) win Kaggle Competitions (by Yanir Seroussi)

Lecture 20: Feature Engineering & Feature Selection

Lecture Slides

• Feature Engineering & Feature Selection

Resources

• ESL Section 3.3: Subset Selection

• Guyon and Elisseeff (2003). An Introduction to Variable and Feature Selection, JMLR.

Lecture 21: Machine Learning Failures and Unintended Harms

Lecture Slides

• ML failures and unintended effects

Resources

• Suresh & Guttag (2020). A Framework for Understanding the Unintended Consequences of Machine Learning. arXiv:1901.10002

• Lehman et al (2019). The Surprising Creativity of Digital Evolution. arXiv:1803.03452

• Kaufman et al. (2012). Leakage in data mining: Formulation, detection and avoidance. ACM Transactions on Knowledge Discovery from Data [pdf].

• Muller et al. (2019). How Data Science Workers Work with Data. Proceedings of the Conference on Human Factors in Computing Systems [pdf].

• Jia et al. (2019). Anthropogenic biases in chemical reaction data hinder exploratory inorganic synthesis. Nature [link] [popular science article].

• AI Incident Database

• Calling Bullsh*t in the Age of Big Data (course developed by C. Bergstrom & J. West @ UW-Seattle) [videos] [website]

  • Case study: Criminal Machine Learning

• fast.ai short course: Practical Data Ethics

• Documentary: Coded Bias (available on PBS and Netflix), directed by Shalini Kantayya.

Lecture 22: In-class activity

Lecture Slides

• Discussion Questions

Resources

• Bender, Gebru et al. (2021). On the Dangers of Stochastic Parrots. FAccT [pdf].

• Loft (2020). Earth system modeling must become more energy efficient. Eos [link].

• Green Algorithms [website] [paper]

Lecture 23: Deep Neural Networks

Lecture Slides

• Neural Networks Part 2: CNNs and GANs

Resources

• The Neural Network Zoo (by Fjodor van Veen)

• Convolutional Neural Networks

  • Image kernels explained visually (by Victor Powell)

  • An Interactive Node-Link Visualization of Convolutional Neural Networks (by Adam W. Harley)

  • Convolutional Neural Networks (CNNs/ConvNets) from CS231N @ Stanford University

  • Convolutional Neural Network cheat sheet (by Afshine Amidi and Shervine Amidi)

  • How do CNNs work?

  • An intuitive explanation of CNNs

  • Explaining TensorFlow Code for a CNN (by Jessica Yung)

• Goodfellow et al. (2014). Generative Adversarial Networks. arXiv:1496.2661

• Reference text: Deep Learning (by Goodfellow, Bengio & Courville)

  • Chapter 6: Deep Feedforward Networks

  • Chapter 9: Convolutional Networks

  • Chapter 20: Deep Generative Models

• Advanced topic: Attention networks

  • Attention and Augmented Recurrent Neural Networks (by Olah & Carter)

  • Attention? Attention! (by Lilian Weng)

Lecture 24: Big Data

Lecture Slides

• Machine Learning and Big Data

Resources

• Reinforcement Learning for Mario Kart [video]

• Self-supervised learning: SEER: The start of a more powerful, flexible, and accessible era for computer vision (Facebook AI blog)

• Settles (2011). From Theories to Queries: Active Learning in Practice [pdf]

  • Overview of Active Learning for Deep Learning (by Jacob Gildenblat)

• Bingham & Mannila (2001). Random projections in Dimensionality Reduction [pdf]

  • Random Projection (sklearn)

  • Seitola et al. (2014). Random Projections in Reducing the Dimensionality of Climate Simulation Data. [link]

• Mining Massive Datasets (by Leskovec, Rajaraman & Ullman): Clustering (Chapter 7) [pdf]

  • Bradley, Fayyad & Reina (1998). Scaling Clustering Algorithms to Large Databases [pdf]