Embedded Files
Week 1: Foundations I
Week 1: Foundations I
Lecture 1: EEPS 1340 Course Overview
Lecture 1: EEPS 1340 Course Overview
Lecture Slides: What is Machine Learning & Course Logistics [Brown login required]
Readings (optional)
[blog post] Training the Next Generation of Physical Data Scientist. McGovern & Allen (2021), Eos.
[perspective] Science and data science. Blei & Smith (2017), PNAS.
[advanced - review] Scientific discovery in the age of AI. Wang et al. (2023), Nature.
[advanced - review] Machine Learning for Geosciences: Challenges and Opportunities. Karpatne et al. (2017).
References
Data-intensive science
Applications of ML in EEPS
[automation] Geomorphological Analysis Using Unpiloted Aircraft Systems, Structure from Motion, and Deep Learning.
[automation] A flexible deep learning crater detection scheme using Segment Anything Model (SAM).
[modeling] A variational LSTM emulator of sea level contribution from the Antarctic ice sheet.
[discovery] Novelty Detection for Multispectral Images with Application to Planetary Exploration.
[discovery] Data-driven discovery of coordinates and governing equations.
Week 2: Foundations II
Week 2: Foundations II
Lecture 2: Introduction to Machine Learning
Lecture 2: Introduction to Machine Learning
Lecture Slides: Intro to Machine Learning
Readings & Resources
[optional] ISL: Chapter 2 (ISL: Introduction to Statistical Learning with Applications in Python -- see Resources page)
[optional] Electoral Precedent (xkcd)
Code example: KNN for Iris Dataset [opens Colab Notebook]
Lecture 3: Mathematical Foundations of Machine Learning
Lecture 3: Mathematical Foundations of Machine Learning
Lecture Slides: Mathematical Foundations of Machine Learning
Pre-lecture activities
[10 minutes] Watch this video ("Vectors, what even are they?") to review the concept of a vector. All students should watch this, even if you believe you are already familiar with the concept of a vector.
Review additional linear algebra topics, as necessary. When deciding which topics to review, ask yourself whether you are familiar with both the "Physics student" view and the "Computer Science student" view of the concept and have the ability to translate between them. [Credit: linear algebra review videos were created by Grant Sanderson]
[10 minutes] Basis vectors, Linear combination, Span, Linear (in)dependence
[11 minutes] Matrices, Matrix-vector multiplication, Linear transformations
[10 minutes] Matrix multiplication
[4 minutes] Rank, Null space (The full video also includes linear systems and matrix inverse -- these are useful concepts to know but not required.)
[4 minutes] Non-square matrix, transformations between dimensions
[2 minutes] Dot product (inner product) (Watch to 2:10 for basic definition of the dot/inner product. The rest of the video (optional, but recommended) covers projections, vectors as matrices/transformations, and duality.)
[13 minutes] Change of basis (Critical concept: a vector (data) can be represented using different choices of basis and it is possible to translate the representation of a vector in one basis to its representation in another.)
[10 minutes] Eigenvectors, Eigenvalues, Eigenbasis (Focus on 0:00-6:00 and 13:04-16:28, and don't worry too much about determinants or computing eigenvectors/values).
If you're not into videos, you are welcome to review the following: Linear Algebra Review and Reference [pdf] by Zico Kolter and Chuong Do, for Stanford University's Machine Learning Course (CS229)
Review calculus topics, as necessary.
[6 minutes] Multivariable functions
[5 minutes] Transformations I; [4 minutes] Transformations II
[5 minutes] Gradient (definition); [6 minutes] Direction of steepest ascent
[8 minutes] Multivariable maxima and minima
Review probability and statistics topics, as necessary.
Seeing Theory interactive visualizations of concepts in probability and statistics
[15 minutes] Ch. 1 Expectation, Variance, Ch. 2 Conditional Probability & Ch. 3 Random Variables -- the other chapters are useful but not required.
Additional resources
[optional] Math for Machine Learning [pdf] (Hal Daumé III, University of Maryland)
[optional] Machine Learning Math Essentials [pdf] (Jeff Howbert, University of Washington)
Week 3: Pattern Discovery I
Week 3: Pattern Discovery I
Lecture 4: Clustering, Part 1
Lecture 4: Clustering, Part 1
Lecture Slides: Clustering Part 1: Hierarchical Clustering
Resources:
Hierarchical clustering example: Soares et al. (2018). The use of hierarchical clustering for the design of optimized monitoring networks, Atmos. Chem. Phys.
ISL: Section 10.3 (Clustering) (ISL: Introduction to Statistical Learning)
ESL: Section 14.3 (Clustering) (ESL: Elements of Statistical Learning)
Lecture 5: Clustering, Part 2
Lecture 5: Clustering, Part 2
Lecture Slides: Clustering Part 2: K-Means
Resources:
scikit-learn Clustering: Covers K-means, Hierarchical clustering, DBSCAN and other clustering algorithms available in sklearn.cluster module.
Visualizing K-Means Clustering interactive demo (by Naftali Harris)
K-means example: Studying Thermal Effluent Impacts in Narragansett Bay using K-means, by former EEPS 1340/1960D undergrad Jonny Benoit and EEPS Prof. Baylor Fox-Kemper.
Week 4: Pattern Discovery II
Week 4: Pattern Discovery II
Lecture 6: Dimensionality Reduction Part 1
Lecture 6: Dimensionality Reduction Part 1
Lecture Slides: DBSCAN & PCA
Resources
DBSCAN
Visualizing DBSCAN Clustering interactive demo (by Naftali Harris)
DBSCAN paper: Ester et al. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise, KDD.
DBSCAN example: Crowdsourced Earthquake Early Warning using ANNs and DBSCAN, by Kong et al.
PCA
ISL Section 10.2 (PCA), ESL Section 14.5 (PCA)
The SVD and Low-Rank Matrix Approximations [pdf] (by Tim Roughgarden & Gregory Valiant, Stanford University, for CS 168)
Lecture 7: Dimensionality Reduction Part 2
Lecture 7: Dimensionality Reduction Part 2
Lecture Slides: Linear Dimensionality Reduction: PCA & NMF
Resources
ISL Section 10.2 (PCA), ESL Section 14.5 (PCA), and ESL Section 14.6 (NMF)
Example (PCA for high-dimensional data): Novembre et al. (2008). Genes Mirror geography within Europe, Nature.
Example (Empirical Orthogonal Functions): Jiang & Zhu (2018).
NMF + K-means example: Clustering seismic waveforms, Holtzman et al. (2018).
NMF paper: Lee & Seung (1999). Learning parts of objects by non-negative matrix factorization, Nature.
Empirical Orthogonal Function (EOF) Analysis (by Dennis Shea, NCAR)
Week 5: Pattern Discovery III
Week 5: Pattern Discovery III
President's Day - No University Exercises
President's Day - No University Exercises
Lecture 8: Non-linear Dimensionality Reduction & Unsupervised Learning Case Study
Lecture 8: Non-linear Dimensionality Reduction & Unsupervised Learning Case Study
Lecture Slides: t-SNE & Unsupervised Learning Case Study
Resources:
How to Use t-SNE Effectively interactive paper (by Wattenberg et al)
t-SNE paper: van der Maaten & Hinton (2008). Visualizing Data using t-SNE, Journal of Machine Learning Research (JMLR).
Case study paper: Sonnewald et al. (2020). Elucidating ecological complexity: Unsupervised learning determines global marine eco-provinces, Science Advances.
Introduction to autoencoders (by Jeremy Jordan)
Self Organizing Maps: Fundamentals [pdf] and Properties and Applications [pdf] (from Neural Computation course notes by John Bullinaria)
Week 6: Predictive Modeling I
Week 6: Predictive Modeling I
Lecture 9: Introduction to Supervised Learning
Lecture 9: Introduction to Supervised Learning
Lecture Slides: Feature Extraction, When Not to Use ML & Intro to Supervised Learning
Resources:
ISL Chapter 2: Statistical Learning
Feature Selection (sklearn docs)
Examples of feature extraction packages: tsfresh & TSFEL (time series); librosa & audioflux (music and audio analysis); pyfeats (image features); sklearn.feature_extraction (text and images); and many domain-specific tools, e.g., MorphoPy (neuroscience)
Lecture 10: Intro to Supervised Learning II & KNN
Lecture 10: Intro to Supervised Learning II & KNN
Lecture Slides: Assessing Model Accuracy & K-Nearest Neighbors
Resources:
K-nearest neighbors example: Lee et al. (2019) A Machine Learning (kNN) Approach to Predicting Global Seafloor Total Organic Carbon.
ISL Sections 2.2.3 (The Classification Setting), 4.6.4 (K-Nearest Neighbors)
Cross-validation code example: KNN for Iris Dataset [opens Colab Notebook]
A Complete Guide to KNN with Applications in Python and R (by Kevin Zakka)
Week 7: Predictive Modeling II
Week 7: Predictive Modeling II
Lecture 11: Linear Regression & Cross-Validation
Lecture 11: Linear Regression & Cross-Validation
Lecture Slides: Cross-validation & Linear Regression
Resources:
ISL Chapter 3: Linear Regression
ISL Section 5.1: Cross-validation
Lecture 12: LASSO & Ridge - Regularization I
Lecture 12: LASSO & Ridge - Regularization I
Lecture Slides: Regularization I
Resources:
ISL Section 6.2: Shrinkage Methods
Chapter 2: The Lasso for Linear Models, in Statistical Learning with Sparsity: The Lasso and Generalizations [pdf] (by Hastie, Tibshirani and Wainwright)
Regularization example: Data-driven discovery of PDEs, Rudy et al. (2017).
Week 8: Predictive Modeling III
Week 8: Predictive Modeling III
Lecture 13: Regularization II
Lecture 13: Regularization II
Lecture Slides: Regularization II
Resources:
Understanding Maximum Likelihood an interactive visualization (by Kristoffer Magnusson)
Statistics: Are you a Bayesian or Frequentist? blog post & video (by Cassie Kozyrkov)
ESL Section 8.2: The Bootstrap and Maximum Likelihood Methods
Lecture 14: Regularization III & Logistic Regression
Lecture 14: Regularization III & Logistic Regression
Lecture Slides: Regularization III & Logistic Regression
Resources:
ISL Section 4.3: Logistic Regression
Logistic regression example: Landslide susceptibility prediction, Rasyid et al. (2016).
Week 9: Predictive Modeling IV
Week 9: Predictive Modeling IV
Lecture 15: Linear Support Vector Classifier
Lecture 15: Linear Support Vector Classifier
Lecture Slides: Support Vector Machines I
Resources:
ISL Chapter 9: Support Vector Machines
Support Vector Machines interactive demo (by Jonas Greitemann)
Lecture 16: Kernel Methods in ML & Support Vector Machine (SVM)
Lecture 16: Kernel Methods in ML & Support Vector Machine (SVM)
Lecture Slides: Support Vector Machines II
Resources:
Lecture Notes (CS 229 @ Stanford): Kernel Methods & Support Vector Machines [pdf] (by Andrew Ng)
ROC curve demonstration (by Alex Rogozhinkov)
Spring Break (no class meetings)
Spring Break (no class meetings)
Week 10: Predictive Modeling V
Week 10: Predictive Modeling V
Lecture 17: Decision Trees I
Lecture 17: Decision Trees I
Lecture Slides: Review of Linear Classifiers & Decision Trees I
Resources:
ISL Chapter 8: Tree-Based Methods
A Visual Introduction to Machine Learning (a scrollable visualization of a decision tree)
Lecture 18: Decision Trees II & Ensemble Methods I
Lecture 18: Decision Trees II & Ensemble Methods I
Lecture Slides: Decision Trees & Random Forests
Resources:
ISL 5.2: The Bootstrap
Seeing Theory: The Bootstrap (visualizing bootstrapping)
Week 11: Predictive Modeling VI
Week 11: Predictive Modeling VI
Lecture 19: Ensemble Methods II & Neural Networks I
Lecture 19: Ensemble Methods II & Neural Networks I
Lecture slides: Boosting & Artificial Neural Networks
Resources:
Gradient Boosting explained (series of demos on trees and boosting)
Gradient Boosting Interactive Playground (interactive demo by same author as above)
Random forest example: Predicting large wildfires in California, Li et al. (2020).
Random forest example: Predicting laboratory earthquakes, Rouet-Leduc et al. (2017).
XGBoost example: Comparison of ML algorithms for maize yield predictions, Kang et al. (2020).
Tinker with a Neural Network in your Browser interactive demo (by Google)
Recommended Videos (by Grant Sanderson) [note that these are sponsored videos containing promotions]
In-class Midterm Exam
In-class Midterm Exam
Week 12: Advanced Topics I
Week 12: Advanced Topics I
Lecture 20: Neural Networks II
Lecture 20: Neural Networks II
Lecture Notes: Training Neural Networks & Convolutional Neural Networks
Resources
The Neural Network Zoo (by Fjodor van Veen)
Convolutional Neural Networks
CNN / Transfer Learning example: Remote Sensing and Poverty Mapping, Xie et al. (2016).
Image kernels explained visually (by Victor Powell)
An Interactive Node-Link Visualization of Convolutional Neural Networks (by Adam W. Harley)
Convolutional Neural Networks (CNNs/ConvNets) from CS231N @ Stanford University
Convolutional Neural Network cheat sheet (by Afshine Amidi and Shervine Amidi)
Explaining TensorFlow Code for a CNN (by Jessica Yung)
Reference text: Deep Learning (by Goodfellow, Bengio & Courville)
Chapter 6: Deep Feedforward Networks
Chapter 9: Convolutional Networks
Chapter 20: Deep Generative Models
Week 13: Advanced Topics II
Week 13: Advanced Topics II
Lecture 22: Practical Tips for Applied ML
Lecture 22: Practical Tips for Applied ML
Lecture Slides: Practical Tips: Debugging, Class Imbalance & Missing Data
Resources
Data and Feature Debugging and Model Debugging (Google Developers Course)
Hyperparameter Optimization (Neural Networks), CS231N Notes, Stanford University
Exploring Bayesian Optimization (Agnihotri & Batra, 2020)
tools: KerasTuner, scikit-optimize
ROC, AUC, precision and recall visually explained (blog post by Paul van der Laken)
Class-imbalanced Data
imbalanced-learn: package for learning from imbalanced data in scikit-learn
Imbalanced data example: Digital soil mapping with SMOTE, Taghizadeh-Mehrjardi et al. (2019).
Missing Data & Imputation
ISL Section 9.6: Missing Data
Imputing Missing Values in scikit-learn
Imputation example: Missing sub-hourly precipitation data, Chivers et al. (2020).
Data leakage
Leakage in data mining: Formulation, detection and avoidance [pdf]. Kaufman et al. (2012)
Leakage and the Reproducibility Crisis in ML-based Science (Kapoor & Narayanan, 2023)
Lecture 23: Physics-informed Machine Learning
Lecture 23: Physics-informed Machine Learning
Lecture Notes: Project Tips & Physics-informed ML
Resources
Kashinath et al. (2021). Physics-informed machine learning: Case studies for weather and climate modeling [Review paper]. Philosophical Transactions of the Royal Society A.
Harder et al. (2022). Generating physically-consistent high-resolution climate data with hard-constrained neural networks. arXiv:2208.05424.
PI-NN example with Code: So what is a physics-informed neural network (blog post by Ben Moseley)
Code: Harmonic Oscillator [Colab notebook]
Week 14: Advanced Topics III
Week 14: Advanced Topics III
Lecture 24: Gaussian Process Regression
Lecture 24: Gaussian Process Regression
Lecture Slides: Gaussian Process Regression
Resources:
Understanding Gaussian Processes & Fitting a Gaussian Process Kernel (by Peter Roelants)
A Visual Exploration of Gaussian Processes (by Görtler et al)
Gaussian Processes, not quite for dummies (by Yuge Shi)
Exploring Bayesian Optimization (Agnihotri & Batra, 2020)
Reference text: Gaussian Processes for Machine Learning [pdfs] by Rasmussen and Williams
GP emulator example: Modeling Greenland Ice Sheet with a GP surrogate, Domingo et al. (2020).
Bayesian Optimization example: Intelligent Environmental Monitoring, Marchant & Ramos (2012).
Thanks to Ethan Kyzivat (grad TA, Spring 2021), Benny Smith (undergrad Data Science Fellow, Spring 21) and Nikolai Stambler (undergrad Data Science Fellow, Fall 2021) for their contributions to the development of course materials for EEPS 1960D (previous course number for EEPS 1340).
Page updated
Report abuse