Lectures

Lecture 1:  Introduction to Data Science & EEPS 1340

Lecture Slides: 

• Intro to Data Science & Course Logistics (pdf)  [Brown login required]

Readings (optional)

• [perspective]  Science and Data Science.

• [blog post]  Training the Next Generation of Physical Data Scientists.

• [Advanced - Review]  Machine Learning for the Geosciences: Challenges and Opportunities. 

• [Advanced - Review]  Machine learning for data-driven discovery in solid Earth geoscience. [full paper]  [extended abstract]  (by Prof. Bergen) 

References:  Machine Learning in EEPS

• • Geomorphological Analysis Using Unpiloted Aircraft Systems, Structure from Motion, and Deep Learning  [Automation] 

  • A variational LSTM emulator of sea level contribution from the Antarctic ice sheet  [Modeling]

  • Novelty Detection for Multispectral Images with Application to Planetary Exploration  [Discovery #1] 

  • Data-driven discovery of coordinates and governing equations  [Discovery #2] 

Python Tutorial

Slides: Tutorial: Python Programming by Yiran  [Brown login required]

Lecture 4:  Clustering, Part 1

Lecture Slides:  Intro to Unsupervised Learning & Cluster Analysis

Resources:

• Hierarchical clustering example:  Soares et al. (2018). The use of hierarchical clustering for the design of optimized monitoring networks,  Atmos. Chem. Phys. 

• [optional]  ISL: Section 10.3 (Clustering)   (ISL: Introduction to Statistical Learning)

• [optional]  ESL: Section 14.3 (Clustering)  (ESL: Elements of Statistical Learning)

Lecture 5:  Clustering, Part 2

Lecture Slides:  Clustering Part 2: K-Means & DBSCAN

Resources

• scikit-learn Clustering: Covers K-means, Hierarchical clustering, DBSCAN and other clustering algorithms available in  sklearn.cluster  module.

• K-means

  • Visualizing K-Means Clustering  interactive demo  (by Naftali Harris)

  • K-means example:  Studying Thermal Effluent Impacts in Narragansett Bay using K-means, by former EEPS 1340/1960D undergrad Jonny Benoit and EEPS Prof. Baylor Fox-Kemper.

• DBSCAN

  • Visualizing DBSCAN Clustering  interactive demo (by Naftali Harris)

  • DBSCAN paper:  Ester et al. (1996).  A density-based algorithm for discovering clusters in large spatial databases with noise, KDD. 

  • DBSCAN example: Crowdsourced Earthquake Early Warning using ANNs and DBSCAN, by Kong et al. 

Lecture 6:  Dimensionality Reduction, Part 1

Lecture Slides:  DBSCAN and PCA

Resources

• ISL Section 10.2 (PCA),  ESL Section 14.5 (PCA)

  • The SVD and Low-Rank Matrix Approximations [pdf]  (by Tim Roughgarden & Gregory Valiant, Stanford University, for CS 168)

Lecture 7:  Dimensionality Reduction, Part 2

Lecture Slides: Dimensionality Reduction - PCA, NMF & t-SNE

Resources

Linear Dimensionality Reduction (PCA & NMF)

• • ISL Section 10.2 (PCA),  ESL Section 14.5 (PCA),  and  ESL Section 14.6 (NMF)

    • Example (PCA for high-dimensional data):  Novembre et al. (2008). Genes Mirror geography within Europe, Nature. 

    • Example (Empirical Orthogonal Functions): Jiang & Zhu (2018). 

    • NMF + K-means example:  Clustering seismic waveforms, Holtzman et al. (2018). 

  • NMF paper:  Lee & Seung (1999).  Learning parts of objects by non-negative matrix factorization, Nature. 

  • Empirical Orthogonal Function (EOF) Analysis (by Dennis Shea, NCAR)

Non-linear Dimensionality Reduction (t-SNE & others)

• • How to Use t-SNE Effectively  interactive paper (by Wattenberg et al) 

    • t-SNE paper:  van der Maaten & Hinton (2008). Visualizing Data using t-SNE, Journal of Machine Learning Research (JMLR). 

  • Introduction to autoencoders (by Jeremy Jordan)

  • Self Organizing Maps: Fundamentals  [pdf] and Properties and Applications [pdf]  (from Neural Computation course notes by John Bullinaria)

Lecture 8:  Unsupervised Learning Case Study

Lecture Slides:  t-SNE, Unsupervised Learning Case Study  & Feature Engineering

Resources

• Videos: Neural Network Autoencode & Self-Organizing Maps  [Brown login]

• Case study paper:  Sonnewald et al. (2020).  Elucidating ecological complexity: Unsupervised learning determines global marine eco-provinces, Science Advances. 

• ESL Section 3.3:  Subset Selection

  • Guyon and Elisseeff (2003).  An Introduction to Variable and Feature Selection, JMLR.

Lecture 9:  Introduction to Supervised Learning

Lecture Slides:  When not to use ML & Intro to Supervised Learning

Resources

• ISL  Chapter 2:  Statistical Learning

• xkcd: Curve Fitting

Lecture 10:  KNN and Linear Regression

Lecture Slides: KNN and Linear Regression

Resources

• K-nearest neighbors example:  Lee et al. (2019)  A Machine Learning (kNN) Approach to Predicting Global Seafloor Total Organic Carbon.

• ISL Sections 2.2.3 (The Classification Setting), 4.6.4 (K-Nearest Neighbors)

• Cross-validation code example: KNN for Iris Dataset  [opens Colab Notebook]

• A Complete Guide to KNN with Applications in Python and R  (by Kevin Zakka)

Lecture 11:  Cross-validation and Regularization I

Lecture Slides:  Cross-Validation, Linear Regression & Regularization

Resources

• ISL Chapter 3:  Linear Regression

• ISL Section 5.1: Cross-validation

Lecture 12:  Regularization II

Lecture Slides:  Ridge, LASSO & Sparsity

Resources

• ISL Section 6.2: Shrinkage Methods

• Chapter 2:  The Lasso for Linear Models, in Statistical Learning with Sparsity: The Lasso and Generalizations [pdf]  (by Hastie, Tibshirani and Wainwright)

• Regularization example: Data-driven discovery of PDEs, Rudy et al. (2017). 

Lecture 13:  Regularization III & Linear Classifiers

Lecture Slides:  Regularization, Maximum Likelihood & Logistic Regression

Resources

• Understanding Maximum Likelihood an interactive visualization (by Kristoffer Magnusson)

• Statistics: Are you a Bayesian or Frequentist?  video + blog post (by Cassie Kozyrkov)

• ESL Section 8.2: The Bootstrap and Maximum Likelihood Methods

• ISL Section 4.3: Logistic Regression

• Logistic regression example: Landslide susceptibility prediction, Rasyid et al. (2016). 

Lecture 14:  Support Vector Machines (SVMs) I

Lecture Slides:  Maximal Margin Classifier and Linear SVM

Resources

• ISL Chapter 9:  Support Vector Machines

• Support Vector Machines interactive demo (by Jonas Greitemann)

Lecture 15:  Support Vector Machines (SVMs) II

Lecture Slides: The Kernel Trick & Support Vector Machines

Resources

• Lecture Notes (CS 229 @ Stanford):  Kernel Methods & Support Vector Machines  [pdf]  (by Andrew Ng)

• ROC curve demonstration (by Alex Rogozhinkov)

Lecture 16:  Decision Trees & Ensemble Methods I

Lecture Slides:  Classification Metrics & Decision Trees

Resources

• ISL Chapter 8: Tree-Based Methods

• A Visual Introduction to Machine Learning (a scrollable visualization of a decision tree)

Lecture 17:  Decision Trees & Ensemble Methods II

Lecture Slides: Decision Trees & Random Forests

Resources

• ISL 5.2: The Bootstrap

• Seeing Theory: The Bootstrap (visualizing bootstrapping)

• Ensemble Learning: Bagging & Boosting

• Gradient Boosting explained (series of demos on trees and boosting)

• Gradient Boosting Interactive Playground (interactive demo by same author as above)

• Random forest example: Predicting large wildfires in California, Li et al. (2020). 

• Random forest example: Predicting laboratory earthquakes, Rouet-Leduc et al. (2017).

• XGBoost example: Comparison of ML algorithms for maize yeild predictions, Kang et al. (2020). 

Lecture 18: Neural Networks I

Lecture Slides:  Ensembles (Boosting) and Artificial Neural Networks 

Resources

• Tinker with a Neural Network in your Browser  interactive demo (by Google)

• Recommended Videos (by Grant Sanderson) [note that these are sponsored videos containing promotions]

• But what is a Neural Network? 

• Gradient descent, how neural networks learn

• What is backpropagation really doing?

Lecture 19: Neural Networks II

Lecture Slides:  Training Neural Networks & Convolutional Neural Networks (CNN)

Resources

• The Neural Network Zoo (by Fjodor van Veen)

• Convolutional Neural Networks

  • • • CNN / Transfer Learning example: Remote Sensing and Poverty Mapping, Xie et al. (2016).

      • Image kernels explained visually (by Victor Powell)

      • An Interactive Node-Link Visualization of Convolutional Neural Networks (by Adam W. Harley)

      • Convolutional Neural Networks (CNNs/ConvNets)  from CS231N @ Stanford University

      • Convolutional Neural Network cheat sheet (by Afshine Amidi and Shervine Amidi)

      • How do CNNs work?

      • An intuitive explanation of CNNs

      • Explaining TensorFlow Code for a CNN (by Jessica Yung)

• NN autoencoder example:  Seamount discovery in bathymetric data, Valentine et al. (2013). 

• Reference text: Deep Learning  (by Goodfellow, Bengio & Courville)

  • • • Chapter 6:  Deep Feedforward Networks

      • Chapter 9:  Convolutional Networks

      • Chapter 20:  Deep Generative Models

Lecture 20:  In-class activity

Slides: see Canvas for link

ML Interactive Demos:

• K-means

• DBSCAN

• t-SNE

• Maximum Likelihood Estimation (MLE)

• Support Vector Machine (SVM)

• ROC Curves

• Decision Tree

• Gradient Boosting

• Neural Networks

Lecture 21:  A Few Useful Things to Know about Machine Learning

Slides: see Canvas for link

Resources

• Domingos, P. (2012). A Few Useful Things to Know About Machine Learning. Communications of the ACM. doi: 10.1145/2347736.2347755  [pdf] [pdf (alternative link)]

Lecture 22:  Neural Networks III

Slides:  Neural Networks III: Review & GANs

Resources

• Goodfellow et al. (2014). Generative Adversarial Networks. arXiv:1496.2661

  • • • Designed to Deceive: Do These People Look Real to You?  NYTimes, published Nov 21, 2020. 

      • GAN example: Super-resolution of Planet CubeSat Images, Lezine, Kyzivat & Smith (2021). 

Lecture 23:  Gaussian Process Regression 

Slides:  Gaussian Processes

Resources

• Generative Modeling: Quadratic Discriminant Analysis (QDA)

  • • • Six Varieties of Gaussian Discriminant Analysis, Math for Machines Blog 

      • Linear & Quadratic Discriminant Analysis, UC Business Analytics R Programming Guide

      • QDA examples:  Tree ring analysis,  Eberhardt et al. (2022)

    • Gaussian Processes (GPs)

      • Understanding Gaussian Processes  &  Fitting a Gaussian Process Kernel  (by Peter Roelants)

      • A Visual Exploration of Gaussian Processes  (by Görtler et al)

      • Gaussian Processes, not quite for dummies  (by Yuge Shi)

      • Exploring Bayesian Optimization  (Agnihotri & Batra, 2020)

      • Reference text:  Gaussian Processes for Machine Learning  [pdfs]  by Rasmussen and Williams

      • GP emulator example:  Modeling Greenland Ice Sheet with a GP surrogate, Domingo et al. (2020).

      • Bayesian Optimization example:  Intelligent Environmental Monitoring, Marchant & Ramos (2012).

Lecture 24: Practical Tips 

Slides:   Tips for Applying ML in Practice (Debugging, Imbalanced Data, Missing Data, Data Leakage)

Resources

• Data and Feature Debugging and  Model Debugging (Google Developers Course)

• Hyperparameter Optimization (Neural Networks), CS231N Notes, Stanford University

  • • • Exploring Bayesian Optimization  (Agnihotri & Batra, 2020)

      • tools: KerasTuner, scikit-optimize

• ROC, AUC, precision and recall visually explained (blog post by Paul van der Laken)

• Class-imbalanced Data

  • • • imbalanced-learn:  package for learning from imbalanced data in scikit-learn

      • Imbalanced data example:  Digital soil mapping with SMOTE, Taghizadeh-Mehrjardi et al. (2019). 

• Missing Data & Imputation

  • • • ISL Section 9.6:  Missing Data

      • Imputing Missing Values in scikit-learn

      • Imputation example:  Missing sub-hourly precipitation data, Chivers et al. (2020). 

    • Data leakage

      • Leakage in data mining: Formulation, detection and avoidance [pdf]. Kaufman et al. (2012)

      • Leakage and the Reproducibility Crisis in ML-based Science (Kapoor & Narayanan, 2023) 

Lecture 25:  Machine Learning Failures and Unintended Effects

Slides:  ML errors, Missing Data, Unintended effects, etc. 

Resources

• A Framework for Understanding the Unintended Consequences of Machine Learning. Suresh & Guttag (2020).  

  • How Data Science Workers Work with Data:  Discovery, Capture, Curation, Design, Creation [pdf]. Muller et al. (2019). 

• The Surprising Creativity of Digital Evolution. Lehman et al (2019).

• Anthropogenic biases in chemical reaction data hinder exploratory inorganic synthesis [link] [popular science article]. Jia et al. (2019).  

• Hundreds of AI tools have been build to catch COVID. None of the helped.  MIT Technology Review (2021). 

• AI Incident Database

Lecture 26: Physics-Informed Machine Learning

Slides:  Sources of Bias & Physics-Informed (Knowledge-Guided) ML

Resources

• Physics-Informed Machine Learning: Case Studies for Weather and Climate Modelling.  Kashinath et al. (2020). 

• Case studies (also see references in review paper above)

  • Enforcing Conservation Laws in NN Climate Emulators.  Beucler et al. (2019) 

  • Physics-constrained Climate Downscaling. Harder et al. (2023)

• PI-NN example with Code:  So what is a physics-informed neural network (blog post by Ben Moseley)

  • Code:  Harmonic Oscillator  [Colab notebook]