Lectures

Lecture 1:  Introduction to Data Science & EEPS 1960D

Lecture Slides: 

• Intro to Data Science & Course Logistics (pdf)  [Brown login required]

Readings (optional)

• [perspective]  Science and Data Science.

• [blog post]  Training the Next Generation of Physical Data Scientists.

• [Advanced - Review]  Machine Learning for the Geosciences: Challenges and Opportunities. 

• [Advanced - Review]  Machine learning for data-driven discovery in solid Earth geoscience. [full paper]  [extended abstract]  (by Prof. Bergen) 

References:  Machine Learning in EEPS

• • Geomorphological Analysis Using Unpiloted Aircraft Systems, Structure from Motion, and Deep Learning [Automation] 

  • Could Machine Learning Break the Convection Parameterization Deadlock?  [Modeling]

  • Novelty Detection for Multispectral Images with Application to Planetary Exploration  [Discovery #1] 

  • Data-driven discovery of coordinates and governing equations [Discovery #2] 

Python Tutorial

• Python Programming and Pitfalls (pdf)  [Brown login required]

Lecture 4:  Clustering  I

Lecture Slides

•  Intro to Unsupervised Learning & Clustering Part 1  [Brown login required]

Resources

• Hierarchical clustering example:  Soares et al. (2018). The use of hierarchical clustering for the design of optimized monitoring networks,  Atmos. Chem. Phys. 

• [optional]  ISL: Section 10.3 (Clustering)   (ISL: Introduction to Statistical Learning)

• [optional]  ESL: Section 14.3 (Clustering)  (ESL: Elements of Statistical Learning)

Lecture 5:  Clustering II

Lecture Slides

• Clustering Part 2:  K-means   [Brown login required] 

Resources

• Visualizing K-Means Clustering  interactive demo  (by Naftali Harris)

• scikit-learn Clustering: Covers K-means, Hierarchical clustering, DBSCAN and other clustering algorithms available in  sklearn.cluster  module.

Lecture 6:  Dimensionality Reduction I

Lectures Slides 

• DBSCAN & Dimensionality Reduction (PCA)   [Brown login required] 

Resources

• Visualizing DBSCAN Clustering interactive demo (by Naftali Harris)

  • DBSCAN paper:  Ester et al. (1996).  A density-based algorithm for discovering clusters in large spatial databases with noise, KDD. 

  • DBSCAN example: Crowdsourced Earthquake Early Warning. Kong et al. (2019).

• ISL Section 10.2 (PCA),  ESL Section 14.5 (PCA)

  • The SVD and Low-Rank Matrix Approximations [pdf]  (by Tim Roughgarden & Gregory Valiant, Stanford University, for CS 168)

  • Example (PCA for high-dimensional data):  Novembre et al. (2008). Genes Mirror geography within Europe, Nature. 

  • Exmple (Empirical Orthogonal Functions): Jiang & Zhu (2018). 

  • Empirical Orthogonal Function (EOF) Analysis (by Dennis Shea, NCAR)

Lecture 7:  Dimensionality Reduction II 

Lectures Slides 

• Dimensionality Reduction: PCA, NMF & t-SNE  [Brown login required]

• [Bonus content]  Neural Network Autoencoders & Self-organizing Maps  [videos - Brown login required]

Resources

• ESL Section 14.6 (NMF)

  • NMF paper:  Lee & Seung (1999).  Learning parts of objects by non-negative matrix factorization, Nature. 

  • NMF + K-means example:  Clustering seismic waveforms, Holtzman et al. (2018). 

• How to Use t-SNE Effectively  interactive paper (by Wattenberg et al) 

  • t-SNE paper:  van der Maaten & Hinton (2008). Visualizing Data using t-SNE, Journal of Machine Learning Research (JMLR). 

• Introduction to autoencoders (by Jeremy Jordan)

• Self Organizing Maps: Fundamentals  [pdf] and Properties and Applications [pdf]  (from Neural Computation course notes by John Bullinaria)

Lecture 8:  Case Study & Intro to Supervised Learning

Lecture Slides

• Case Study, Feature Extraction & Intro to Supervised Learning   [Brown login required]

Resources

• Case study paper:  Sonnewald et al. (2020).  Elucidating ecological complexity: Unsupervised learning determines global marine eco-provinces, Science Advances. 

• ESL Section 3.3:  Subset Selection

  • Guyon and Elisseeff (2003).  An Introduction to Variable and Feature Selection, JMLR.

• ISL  Chapter 2:  Statistical Learning

Lecture 9:  Supervised Learning II & KNN 

Lecture Slides

• Supervised Learning & KNN  [Brown login required]

Resources

• xkcd: Curve Fitting

• K-nearest neighbors example:  Lee et al. (2019)  A Machine Learning (kNN) Approach to Predicting Global Seafloor Total Organic Carbon.

• ISL Sections 2.2.3 (The Classification Setting), 4.6.4 (K-Nearest Neighbors)

• Cross-validation code example: KNN for Iris Dataset  [opens Colab Notebook]

• A Complete Guide to KNN with Applications in Python and R  (by Kevin Zakka)

Lecture 10:  Cross-validation & Linear Regression

Lecture Slides

• Cross-validation and Linear Regression  [Brown login required]

Resources

• ISL Chapter 3:  Linear Regression

• ISL Section 5.1: Cross-validation

Lecture 11:  Regularization: Ridge & LASSO

Lecture Slides

• Regularization I  [Brown login required]

Resources

• ISL Section 6.2: Shrinkage Methods

• Chapter 2:  The Lasso for Linear Models, in Statistical Learning with Sparsity: The Lasso and Generalizations [pdf]  (by Hastie, Tibshirani and Wainwright)

• Regularization example: Data-driven discovery of PDEs, Rudy et al. (2017). 

Lecture 12:  Regularization II

Lecture Slides

• Sparsity, MLE & Bayesian Linear Regression  [Brown login required]

• Derivation:  Bayesian LASSO

Resources

• Understanding Maximum Likelihood an interactive visualization (by Kristoffer Magnusson)

• Statistics: Are you a Bayesian or Frequentist?  video + blog post (by Cassie Kozyrkov)

• ESL Section 8.2: The Bootstrap and Maximum Likelihood Methods

Lecture 13:  Linear Classifiers

Lecture Slides

• Logistic Regression & SVM I   [Brown login required]

Resources

• ISL Section 4.3: Logistic Regression

• Logistic regression example: Landslide susceptibility prediction, Rasyid et al. (2016). 

Lecture 14:  Support Vector Machines (SVMs) 

Lecture Slides

• Support Vector Machines II   [Brown login required]

Resources

• ISL Chapter 9:  Support Vector Machines

• Support Vector Machines interactive demo (by Jonas Greitemann)

• Lecture Notes (CS 229 @ Stanford):  Kernel Methods & Support Vector Machines  [pdf]  (by Andrew Ng)

• ROC curve demonstration (by Alex Rogozhinkov)

Lecture 15:  Decision Trees & Ensemble Methods I

Lecture Slides

• Decision Trees  [Brown login required]

Resources

• ISL Chapter 8: Tree-Based Methods

• A Visual Introduction to Machine Learning (a scrollable visualization of a decision tree)

Lecture 16:  Decision Trees & Ensemble Methods II

Lecture Slides

• Ensemble Methods: Bagging, Random Forests and Boosting  [Brown login required]

Resources

• ISL 5.2: The Bootstrap

• Seeing Theory: The Bootstrap (visualizing bootstrapping)

• Ensemble Learning: Bagging & Boosting

• Gradient Boosting explained (series of demos on trees and boosting)

• Gradient Boosting Interactive Playground (interactive demo by same author as above)

• Random forest example: Predicting large wildfires in California, Li et al. (2020). 

• Random forest example: Predicting laboratory earthquakes, Rouet-Leduc et al. (2017).

• XGBoost example: Comparison of ML algorithms for maize yeild predictions, Kang et al. (2020). 

Lecture 17:  Neural Networks I

Lecture Slides

• Artificial Neural Networks   [Brown login required]

Resources

• Tinker with a Neural Network in your Browser  interactive demo (by Google)

• Recommended Videos (by Grant Sanderson) [note that these are sponsored videos containing promotions]

• But what is a Neural Network? 

• Gradient descent, how neural networks learn

• What is backpropagation really doing?

Lecture 18:  Neural Networks II

Lecture Slides

• Deep Neural Networks: CNNs, GANs and Autoencoders  [Brown login required]

Resources

• The Neural Network Zoo (by Fjodor van Veen)

• Convolutional Neural Networks

  • 1. • CNN / Transfer Learning example: Remote Sensing and Poverty Mapping, Xie et al. (2016).

       • Image kernels explained visually (by Victor Powell)

       • An Interactive Node-Link Visualization of Convolutional Neural Networks (by Adam W. Harley)

       • Convolutional Neural Networks (CNNs/ConvNets)  from CS231N @ Stanford University

       • Convolutional Neural Network cheat sheet (by Afshine Amidi and Shervine Amidi)

       • How do CNNs work?

       • An intuitive explanation of CNNs

       • Explaining TensorFlow Code for a CNN (by Jessica Yung)

• Goodfellow et al. (2014). Generative Adversarial Networks. arXiv:1496.2661

  • 1. • Designed to Deceive: Do These People Look Real to You?  NYTimes, published Nov 21, 2020. 

       • GAN example: Super-resolution of Planet CubeSat Images, Lezine, Kyzivat & Smith (2021). 

• NN autoencoder example:  Seamount discovery in bathymetric data, Valentine et al. (2013). 

• Reference text: Deep Learning  (by Goodfellow, Bengio & Courville)

  • 1. • Chapter 6:  Deep Feedforward Networks

       • Chapter 9:  Convolutional Networks

       • Chapter 20:  Deep Generative Models

Lecture 19:  Semi-supervised Learning  &  Gaussian Processes I

Lecture Slides

• Generative Models, Semi-Supervised Learning & Gaussian Processes I  [Brown login required]

Resources

• Quadratic Discriminant Analysis

  • Six Varieties of Gaussian Discriminant Analysis, Math for Machines Blog 

  • Linear & Quadratic Discriminant Analysis, UC Business Analytics R Programming Guide

  • QDA examples:  Tree ring analysis,  Eberhardt et al. (2022);  Detection of reduced sulfur on Mars, Wong et al. (2020). 

• Beyond Supervised vs. Unsupervised Learning 

  • Generative modeling example:  DALL-E 2 (OpenAI) 

  • Active learning example:  Species Identification in Camera Trap Images, Norouzzadeh et al. (2019).

  • Weakly supervised learning example:  Cropland Segmentation in Remote Sensing Images, Wang et al. (2020).

  • One-shot learning example:  Acoustic identification of bird species, Acconcjaioco & Ntalampiras (2021).

  • Self-supervised learning examples:  Blind denoising of DAS waveforms, Van den Ende et al. (2021).

• Understanding Gaussian Processes  &  Fitting a Gaussian Process Kernel  (by Peter Roelants)

Lecture 20:  Gaussian Processes II

Lecture Slides

• Gaussian Processes II   [Brown login required]

Resources

• A Visual Exploration of Gaussian Processes  (by Görtler et al)

• Gaussian Processes, not quite for dummies  (by Yuge Shi)

• Exploring Bayesian Optimization  (Agnihotri & Batra, 2020)

• Reference text:  Gaussian Processes for Machine Learning  [pdfs]  by Rasmussen and Williams

• GP emulator example:  Modeling Greenland Ice Sheet with a GP surrogate, Domingo et al. (2020).

• Bayesian Optimization example:  Intelligent Environmental Monitoring, Marchant & Ramos (2012).

Lecture 21:  Discussion & ML Review

Lecture Slides

• Review: A Few Useful Things to Know About ML  [Brown login required]

Pre-class reading

• Domingos (2012),  A few useful things to know about machine learning. [pdf]

Lecture 22:  Practical Tips for Applied Machine Learning

Lecture Slides

• Debugging, Metrics, Imbalanced Data, Missing Data, and more...  [Brown login required]

Resources

• Data and Feature Debugging and  Model Debugging (Google Developers Course)

• Hyperparameter Optimization (Neural Networks), CS231N Notes, Stanford University

  • Exploring Bayesian Optimization  (Agnihotri & Batra, 2020)

  • tools: KerasTuner, scikit-optimize

• ROC, AUC, precision and recall visually explained (blog post by Paul van der Laken)

• Class-imbalanced Data

  • imbalanced-learn:  package for learning from imbalanced data in scikit-learn

  • Imbalanced data example:  Digital soil mapping with SMOTE, Taghizadeh-Mehrjardi et al. (2019). 

• Missing Data & Imputation

  • ISL Section 9.6:  Missing Data

  • Imputing Missing Values in scikit-learn

  • Imputation example:  Missing sub-hourly precipitation data, Chivers et al. (2020). 

Lecture 23:  Physics-Informed ML

Lecture Slides

• Knowledge-guided (physics-informed) ML  [Brown login required]

Resources

• Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data.  Karpatne et al. (2017). 

• Physics-Informed Machine Learning: Case Studies for Weather and Climate Modelling.  Kashinath et al. (2020). 

• Integrating Scientific Knowledge with ML for Engineering and Environmental Systems.  Willard et al. (2020).

• Case studies (from references above)

  • PI-NN example:  Enforcing Conservation Laws in NN Climate Emulators.  Beucler et al. (2019) 

  • Hybrid ML-Modeling example:  Residual Modeling for Numerical Weather Prediction.  Bonavita & Laloyaux (2020). 

  • Interpretability example:  Spatial Analysis of Severe Hailstorms. Gagne et al. (2019). 

  • Stochasticity & Uncertainty example:  Snow water equivalent modeling with a cGAN.  Manepalli et al. (2019).

  • Theory-guided Initialization example:  Imputation of Plant Traits. Schrodt et al. (2015).  

• PI-NN example with Code:  So what is a physics-informed neural network (blog post by Ben Moseley)

  • Code:  Harmonic Oscillator  [Colab notebook] 

Lecture 24:  Machine Learning Failures & Unintended Effects

Lecture Slides

• ML Failures, Sources of Bias, Data leakage, etc.  [Brown login required]

Resources

• A Framework for Understanding the Unintended Consequences of Machine Learning. Suresh & Guttag (2020).  

  • How Data Science Workers Work with Data:  Discovery, Capture, Curation, Design, Creation [pdf]. Muller et al. (2019). 

• The Surprising Creativity of Digital Evolution. Lehman et al (2019).

• Leakage in data mining: Formulation, detection and avoidance [pdf]. Kaufman et al. (2012).

• Anthropogenic biases in chemical reaction data hinder exploratory inorganic synthesis [link] [popular science article]. Jia et al. (2019).  

• Hundreds of AI tools have been build to catch COVID. None of the helped.  MIT Technology Review (2021). 

• AI Incident Database

• Calling Bullsh*t in the Age of Big Data (course developed by C. Bergstrom & J. West @ UW-Seattle) [videos] [website]

  • Case study:  Criminal Machine Learning

• fast.ai short course: Practical Data Ethics

• Documentary: Coded Bias  (available on PBS and Netflix), directed by Shalini Kantayya.

Lectures 25 & 26:  Group Discussions on Ethics, Sustainability & Computation

Slides

• AI & Sustainability   [Brown login required]

• Group discussion summary  [Google Slides -- Brown login required]

Pre-class Reading

• Earth Systems Modeling Must Become More Energy Efficient. EoS. 

• Lannelongue et al. (2021).   Ten simple rules to make your computing more environmentally sustainable.  PLoS Computational Biology.  

• Bender, Gebru et al. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?  ACM Conference on Fairness, Accountability and Transparency. 

Resources

• Harnessing Artificial Intelligence for the Earth.  World Economic Forum, PwC and Stanford University [pdf]. 

• UN Sustainable Development Goals 

  • Climate change example:  Mapping and assessing carbon stock of tropical peatlands. Rudiyanto et al. (2018).

  • Biodiversity & conservation example:  Drones Planting Trees: An Interview with BioCarbon Engineering. Medium. 

  • Disaster resilience example:  Expanding our ML-based flood forecasting. Google Blog.

Link to Additional Resources  (opens new page)

Discussion Questions (Homework 4)