Links

Courses and Learning Resources (ML / AI / Deep Learning)

- List of advanced ML/AI courses from Reddit (Mar 2020, updated Jan 2021)

https://www.reddit.com/r/MachineLearning/comments/fdw0ax/d_advanced_courses_update/

- A curated list of Artificial Intelligence (AI) courses, books, video lectures and papers

https://medium.com/datadriveninvestor/a-curated-list-of-artificial-intelligence-ai-courses-books-video-lectures-and-papers-d2f584ca14f8

- Hacker's guide to Neural Networks (Andrej Karpathy)

http://karpathy.github.io/neuralnets/

- CS231n Convolutional Neural Networks for Visual Recognition (Stanford)

http://cs231n.github.io

- CS230 Deep Learning (Stanford)

https://cs230.stanford.edu/

https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-convolutional-neural-networks (Cheetsheets by Afshine & Shervine Amidi)

- CS221 Artificial Intelligence: Principles and Techniques (Stanford)

http://web.stanford.edu/class/cs221/ (lecture slides available at the bottom of this link)

https://stanford.edu/~shervine/teaching/cs-221 (Cheetsheets by Afshine & Shervine Amidi)

Former student: "The course covers fundamental AI concepts such as regression, clustering, gradient descent, nearest neighbors, path finding as well as basic RL concepts such as Monte Carlo, SARSA, Q-learning, policy/value iteration, etc."

- AA228 / CS238 Decision Making Under Uncertainty (Stanford, notes not publicly available)

http://web.stanford.edu/class/aa228/

- 6.S191: Introduction to Deep Learning (MIT, Alexander Amini & Ava Soleimany)

http://introtodeeplearning.com/

- CS 188 | Introduction to Artificial Intelligence

https://inst.eecs.berkeley.edu/~cs188/fa18

- Dive into Deep Learning (Berkeley Course)

http://d2l.ai/chapter_introduction/index.html

- CS 285 | Deep Reinforcement Learning (UC Berkeley Course)

http://rail.eecs.berkeley.edu/deeprlcourse/

- Introduction to Machine Learning for Coders (Jeremy Howard; from Kaggle and Enlitic)

https://course.fast.ai/ml; blog about the course's release: https://www.fast.ai/2018/09/26/ml-launch/

- Machine Learning Crash Course (Google)

https://developers.google.com/machine-learning/crash-course

- ML Course

https://mlcourse.ai

- Deep Learning with TensorFlow 2 and Keras - Notebooks

https://github.com/ageron/tf2_course

GANs and Generative Modelling

- State-of-the-Art Image Generative Models (Aran Komatsuzaki, Mar 4, 2021)

https://arankomatsuzaki.wordpress.com/2021/03/04/state-of-the-art-image-generative-models/

Collation of some recent SotA image generative models, with short summaries, visualizations, and code (when available). Includes VAEs, GANs, Diffusion Models. AB: Note that since March 2021, new techniques have come out (e.g. StyleGAN3).

A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications

https://arxiv.org/abs/2001.06937

"An absurdly well referenced review of GANs."

ThisEmoteDoesNotExist: Training a GAN for Twitch Emotes

https://blog.twitch.tv/thisemotedoesnotexist-training-a-gan-for-twitch-emotes-a742b6354b73

Discussion on implementing GANs for producing "twitch emotes".

Photos from Crude Sketches: NVIDIA's GauGAN Explained Visually (Adam D King, Apr 11, 2019)

https://adamdking.com/blog/gaugan/

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

https://arxiv.org/abs/1811.10597

Generative Adversarial Networks - The Story So Far (highlights over the past 5 years) (Ajay Uppili Arasanipalai, 21 Jun, 2019)

https://blog.floydhub.com/gans-story-so-far

The Rise of Generative Adversarial Networks #GANs (Kailash Ahirwar, 28 Mar 2019)

https://blog.usejournal.com/the-rise-of-generative-adversarial-networks-be52d424e517

Annotated implementations + introductions for various algorithms including GANs & VAEs

https://github.com/shayneobrien/generative-models

Mode collapse in GANs (Jan 18, 2017)

http://aiden.nibali.org/blog/2017-01-18-mode-collapse-gans/

Variational Autoencoders Explained

http://kvfrans.com/variational-autoencoders-explained/

Nice intro to VAEs

- Intuitively Understanding Variational Autoencoders (Irhum Shafkat, Feb 5, 2018)

https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf

Simple intuitive explanation

Reinforcement Learning

- Implementation of Reinforcement Learning Algorithms (Denny Britz)

http://www.wildml.com/2016/10/learning-reinforcement-learning/

https://github.com/dennybritz/reinforcement-learning

- Training a Donkey Car to Drive using Unity Simulator and Reinforcement Learning (Felix Yu, Sep 11, 2018)

https://flyyufelix.github.io/2018/09/11/donkey-rl-simulation.html

- Reinforcement Learning: a comprehensive introduction [Part 0] (Luca Palmieri, May 11, 2018)

https://www.lpalmieri.com/posts/rl-introduction-00

Deep Learning Hardware (GPUs)

- Choosing the Best GPU for Deep Learning in 2020 (Lambda Labs, Feb 18, 2020)

https://lambdalabs.com/blog/choosing-a-gpu-for-deep-learning

- Hardware for Deep Learning. Part 3: GPU (Grigory Sapunov, Mar 15, 2018)

https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-8906c1644664

- Which GPU(s) to Get for Deep Learning

https://timdettmers.com/2019/04/03/which-gpu-for-deep-learning

- Training ANNs on the Raspberry Pi 4 and Jetson Nano (Romilly Cocking, 14 Jul 2019)

https://blog.rareschool.com/2019/07/training-anns-on-raspberry-pi-4-and.html

"Nano is between 2.8 and 5.7 times faster than the Pi 4 when training"

Understanding Convolution & Deconvolution

- Deconvolution and Checkerboard Artifacts (Oct 17, 2016)

http://distill.pub/2016/deconv-checkerboard/

- Up-sampling with Transposed Convolution (Nov 13, 2017)

"Up-sampling transpose convolutions / Fractionally-strided convolution / Deconvolution"

https://towardsdatascience.com/up-sampling-with-transposed-convolution-9ae4f2df52d0

- Deconvolutional Layers (Stack Exchange answer)

http://datascience.stackexchange.com/questions/6107/what-are-deconvolutional-layers

- Excellent ArXiv papers:

"Understanding Convolution for Semantic Segmentation" https://arxiv.org/pdf/1702.08502.pdf

"A guide to convolution arithmetic for deep learning" https://arxiv.org/pdf/1603.07285.pdf

"Is the deconvolution layer the same as a convolutional layer?" https://arxiv.org/pdf/1609.07009.pdf

Bayesian Neural Networks

A Sober Look at Bayesian Neural Networks (Carles Gelada and Jacob Buckman, 17 Jan, 2020)

https://jacobbuckman.com/2020-01-17-a-sober-look-at-bayesian-neural-networks/

3D Bayesian Convolutional Neural Network (BCNN) for Uncertainty Quantification in Volumetric Segmentation

Paper: https://arxiv.org/abs/1910.10793

Code: https://github.com/sandialabs/bcnn

Coding

TinyML projects

https://www.edgeimpulse.com/blog/a-big-farewell-to-2021-with-21-tinyml-projects

21 different projects using TinyML. e.g. Monitor reckless driving in your neighborhood with a portable speed trap; Spot intruders using ultra-low-powered thermal vision; Measure car tire pressure with machine vision;

AugLy

https://github.com/facebookresearch/AugLy

Data augmentations library. Four modalities (audio, image, text & video) and over 100 augmentations

Beginner's NN pipeline tutorial

https://machinelearningmastery.com/modeling-pipeline-optimization-with-scikit-learn/

Knowledge Distillation Library (KD_Lib)

https://github.com/SforAiDl/KD_Lib

A PyTorch Knowledge Distillation library for benchmarking and extending works in the domains of Knowledge Distillation, Pruning, and Quantization.

Neural Prophet: Neural Network based Time-Series model.

https://github.com/ourownstory/neural_prophet

Tensorflow Object Counting API

https://github.com/ahmetozlu/tensorflow_object_counting_api

A recipe for training neural networks (Karpathy, Apr 2019)

https://karpathy.github.io/2019/04/25/recipe/

Clarifying exceptions and visualizing tensor operations in deep learning code (Terence Parr, Oct 2020)

https://explained.ai/tensor-sensor/index.html

Describes "TensorSensor" code to visualise dimensions of tensors & give more info on exceptions, to help with debugging

Troubleshooting Convolutional Neural Networks

https://gist.github.com/zeyademam/0f60821a0d36ea44eef496633b4430fc

Deep Learning Models

https://github.com/rasbt/deeplearning-models

A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Hypertunity: a hyperparameter optimization library

https://github.com/gdikov/hypertunity

ASFF: Learning Spatial Fusion for Single-Shot Object Detection

https://github.com/ruinmessi/ASFF

Other

Transformers from Scratch

https://e2eml.school/transformers.html

step-by-step, a from-scratch implementation of a Transformer

How IBM’s Deep Blue Beat World Champion Chess Player Garry Kasparov

https://spectrum.ieee.org/the-institute/ieee-history/how-ibms-deep-blue-beat-world-champion-chess-player-garry-kasparov

Made with ML (resources for various topics in ML, e.g. tutorials, code)

https://madewithml.com/topics/

An Epidemic of AI Misinformation (Gary Marcus, Dec 2019)

https://thegradient.pub/an-epidemic-of-ai-misinformation/

Key trends from NeurIPS 2019 (Dec, 2019)

https://huyenchip.com/2019/12/18/key-trends-neurips-2019.html

Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks (Nov, 2019)

[An Alternative to Batch Norm]

https://arxiv.org/abs/1911.09737

In this paper we propose the Filter Response Normalization (FRN) layer, a novel combination of a normalization and an activation function, that can be used as a drop-in replacement for other normalizations and activations. Our method operates on each activation map of each batch sample independently, eliminating the dependency on other batch samples or channels of the same sample. Our method outperforms BN and all alternatives in a variety of settings for all batch sizes.

State of AI report 2019 (Nathan Benaich & Ian Hogarth, 25 Jun 2019)

https://www.slideshare.net/StateofAIReport/state-of-ai-report-2019-151804430

Where We See Shapes, AI Sees Textures (Jordana Cepelewicz, 1 Jul 2019)

https://www.quantamagazine.org/where-we-see-shapes-ai-sees-textures-20190701/

Based on ICLR 2019 paper: "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness", Geirhos et al. https://openreview.net/forum?id=Bygh9j09KX

Time series forecasting

https://github.com/ageron/handson-ml2/blob/master/15_processing_sequences_using_rnns_and_cnns.ipynb

Monte Carlo methods - Why it's a bad idea to go to the casino (Christof Kaser)

https://easylang.online/apps/tutorial_mcarlo.html

Microsoft's alternative to TensorBoard: https://github.com/microsoft/tensorwatch

What is Geometric Deep Learning? (Flawnson Tong, 18 Apr, 2019)

https://medium.com/@flawnsontong1/what-is-geometric-deep-learning-b2adb662d91d

NVIDIA DRIVE™ Labs

https://www.nvidia.com/en-us/self-driving-cars/drive-labs/

Cool videos of various driving DNNs (detecting occlusions of the windscreen, intersections, centre path & lane lines; predicting future object positions)

Why Deep Learning Has Not Superseded Traditional Computer Vision (Zbigniew Zdziarski, Mar 9, 2018)

http://zbigatron.com/has-deep-learning-superseded-traditional-computer-vision-techniques/

Deep learning needs big data; Deep learning is sometimes overkill; Traditional CV will help you with deep learning
(fails to mention Transfer Learning / fine-tuning can overcome the lack of data)

Understanding SVD (Singular Value Decomposition)

https://towardsdatascience.com/svd-8c2f72e264f

This is how AI bias really happens—and why it’s so hard to fix

https://www.technologyreview.com/s/612876/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix/

Huge Collection of Academic Datasets, Papers & Courses

http://academictorrents.com/

Neural network interpretability

https://thegradient.pub/interpretability-in-ml-a-broad-overview/ "Interpretability in ML: A Broad Overview", Nov 2020

https://github.com/davidbau/dissect "Dissect: Understanding the Role of Individual Units in a Deep Network", Sep 2020

https://distill.pub/2018/building-blocks "Building blocks of interpretability"

https://github.com/tensorflow/lucid "Lucid - tensorflow code"

An overview of gradient descent optimization algorithms (Sebastian Ruder)

http://ruder.io/optimizing-gradient-descent/

A birds-eye view of optimization algorithms (Fabian Pedregosa, Apr 3, 2018)

http://fa.bianp.net/teaching/2018/eecs227at/

Includes interactive visualisations

Visualizing Gradient Descent with Momentum in Python (Henry Chang, Aug 13, 2018)

https://medium.com/@hengluchang/visualizing-gradient-descent-with-momentum-in-python-7ef904c8a847

An Insider’s Guide to Keeping Up with the AI Experts (Mat Leonard, Sep 17, 2018)

https://blog.udacity.com/2018/09/machine-learning-ai-experts-on-twitter.html

18 Tips for Training your own Tensorflow.js Models in the Browser: Training efficient Image Classifiers and Object Detectors for the Web with Tensorflow.js (Vincent Mühler, Oct 2, 2018)

https://itnext.io/18-tips-for-training-your-own-tensorflow-js-models-in-the-browser-3e40141c9091

- Depthwise separable convolutions; Skip Connections & Densely Connected Blocks; ReLU activations; Adam optimizer; Weight init (biases = zero; weidths = normal dist or Glorot normal dist); Shuffle inputs; Save model checkpoints
- Check your input data, pre- & post processing logic; Check loss fcn; Overfit on a small dataset first

Glossary of Machine Learning Terms

https://semanti.ca/blog/?glossary-of-machine-learning-terms

Deep Learning for Classifying Hotel Aesthetics Photos (Christopher Lennan and Dat Tran, Oct 30, 2018)

https://devblogs.nvidia.com/deep-learning-hotel-aesthetics-photos

- Automatically assess quality of images (aesthetic & technical dimensions); can successfully rank images
- Transfer learning is great way to bootstrap, but domain specific data proved to be the key to improve results; not much additional data is needed to fine-tune the model
- Earth Mover's Loss
- Code: https://github.com/idealo/image-quality-assessment

AdamW and Super-convergence is now the fastest way to train neural nets (Sylvain Gugger and Jeremy Howard, Jul 2, 2018)

"L2 regularization and weight decay are different when considering SGD+momentum or more sophisticated optimizers like Adam"

http://www.fast.ai/2018/07/02/adam-weight-decay/

Medical Image Segmentation [Part 1] — UNet: Convolutional Networks with Interactive Code

https://towardsdatascience.com/medical-image-segmentation-part-1-unet-convolutional-networks-with-interactive-code-70f0f17f46c6

Using variational autoencoders to learn variations in data

https://news.sophos.com/en-us/2018/06/15/using-variational-autoencoders-to-learn-variations-in-data/

ML beyond Curve Fitting: An Intro to Causal Inference and do-Calculus

http://www.inference.vc/untitled/

AI and Deep Learning in 2017 – A Year in Review

http://www.wildml.com/2017/12/ai-and-deep-learning-in-2017-a-year-in-review

Neural Network Architectures

https://medium.com/towards-data-science/neural-network-architectures-156e5bad51ba

ResNets, HighwayNets, and DenseNets, Oh My!

https://chatbotslife.com/resnets-highwaynets-and-densenets-oh-my-9bb15918ee32

Real-time object detection with YOLO (You Only Look Once)

http://machinethink.net/blog/object-detection-with-yolo

Google's object detection API

https://research.googleblog.com/2017/06/supercharge-your-computer-vision-models.html

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

https://research.googleblog.com/2017/06/mobilenets-open-source-models-for.html

https://hackernoon.com/creating-insanely-fast-image-classifiers-with-mobilenet-in-tensorflow-f030ce0a2991

"instead of a single 3x3 convolution layer followed by batch norm and ReLU, MobileNets split the convolution into a 3x3 depthwise conv and a 1x1 pointwise conv"

Generative Models

https://openai.com/blog/generative-models/

Kullback-Leibler Divergence

https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained

Dataset labelling with humans in the loop

http://www.yf.io/p/lsun