Course Description
This is an introductory course on deep learning methods with applications to computer vision, natural language processing, and more! Students will gain foundational knowledge of deep learning algorithms and get practical experience in building neural networks.
Prerequisites: Calculus (i.e. taking derivatives),linear algebra (i.e. matrix multiplication), Basic Python Programming skill.
Tools Used: Tensrflow & PyTorch
Unit I (Course Material)
Introduction to ANN, Models of a Neuron, Activation functions, Learning and Training: Hebbian, Memory based, Competitive, Supervised and Unsupervised learning, Memory models, Recall and Adaptation, Network Architectures, Single-layered Feed-forward Networks, Multi-layered Feedforward Networks, gradient descent and contemporary variants, back-propagation algorithm, regularization, batch normalization, loss functions, Unit saturation, aka the vanishing gradient problem, and ways to mitigate it. Relu Heuristic s for avoiding bad local minima, Heuristics for faster training, Regularization, Dropout, Recurrent Networks, Topologies.
Introduction of Artificial Neural Network (PPT/PDF)
McCulloch-Pitts Threshold Logic (PPT/PDF)
Hebb Nets, Perceptron's, and Adaline Nets (PPT/PDF)
Gradient Descent (PPT/PDF)
Back Propagation Concept (PPT/PDF) (Example PPT/PDF)
Activation Functions (PPT/PDF)
Loss Function Cross-Entropy loss (PPT)
Regularization/Dropout (PPT/PDF)
Code: Basic Code Pack(Tensorflow) | A Neural Network Playground (tensorflow.org)
BOOKS:- Neural Networks and Learning Machines, 3d Edition https://dai.fmph.uniba.sk/courses/NN/haykin.neural-networks.3ed.2009.pdf
Unit-2 Convolutional Neural Networks
Convolutional Networks- Fundamentals, architectures, pooling, visualization, popular convnet architectures - AlexNet, ZFNet, VGG, C3 D, GoogLeNet, ResNet, MobileNet -v1, Inception, Training a Convnet: weights initialization, batch normalization, hyperparameter optimization. Recurrent Neural Networks LSTM, GRU, Encoder Decoder architectures.
Slides
hyperparameter optimization
Resources for LSTM
https://dennybritz.com/posts/wildml/recurrent-neural-networks-tutorial-part-1/
https://dennybritz.com/posts/wildml/recurrent-neural-networks-tutorial-part-3/
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Demos:-
ConvNET Demo https://cs.stanford.edu/people/karpathy/convnetjs/demo/cifar10.html
LeNET-5 Demo:-http://yann.lecun.com/exdb/lenet/index.html
Unit-3 Deep Unsupervised Networks
Autoencoders (standard, sparse, denoising, contractive, etc), Variational Autoencoders, Adversarial Generative Networks, Autoencoder and DBM.
Autoencoders (standard, sparse, denoising, contractive, etc)
Autoencoders (standard, sparse, denoising, contractive, etc)
Adversarial Generative Networks Minor Change Version
Unit-4 Deep Learning Tools And Applications
Deep Learning Tools: TensorFlow Caffe, Theano, Torch, etc. Case study and applications in Image Processing, Natural Language Processing, Speech Recognition, Video Analytic etc. using different deep neural networks.
Deep Learning Tools: TensorFlow Caffe, Theano, Torch, etc.
case study and applications in Image Processing (Part1)
Speech Recognition, Video Analytic (Part2)
Reference Books:
Ian Goodfellow and Yoshua Bengio and Aaron Courville. Deep Learning. An MIT Press book. 2016.
Dive into Deep Learning ( By Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola)
Neural Networks and Deep Learning (By Michael Nielsen)
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications : https://arxiv.org/pdf/1704.04861.pdf
Deep Residual Learning for Image Recognition https://arxiv.org/pdf/1512.03385.pdf
Identity Mappings in Deep Residual Networks https://arxiv.org/pdf/1603.05027.pdf
Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift :-https://arxiv.org/pdf/1502.03167.pdf
Xavier Glorot Yoshua Bengio -Weight Initialization
Understanding the difficulty of training deep feedforward neural networks
Quick / Easy Reads:
The Self-Assembling Brain: How Neural Networks Grow Smarter, Peter Robin Hiesinger (1st Edition), 2021, Princeton University Press
Behind Deep Blue: Building the Computer That Defeated the World Chess Champion, JFeng-hsiung Hsu (2nd Edition), 2002 / 2022, Princeton University Press [ Video: Deep Blue | Down the Rabbit Hole ]
AI Superpowers: China, Silicon Valley, and the New World Order, Kai-Fu Lee (1st Edition), 2018, Houghton Mifflin Harcour [ Video Lecture ]
AI 2041: Ten Visions for Our Future, Kai-Fu Lee, Chen Qiufan (1st Edition), 2021, Currency
Life 3.0: Being Human in the Age of Artificial Intelligence, Max Tegmark (1st Edition), 2018, Vintage [ Video Lecture ]
Superintelligence: Paths, Dangers, Strategies, Nick Bostrom (1st Edition), 2014, Oxford University Press [ Video Lecture ]
The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind, Marvin Minsky (1st Edition), 2006, Simon & Schuster (Free Book)
The Society of Mind, Marvin Minsky (1st Edition), 1988, Simon & Schuster [ Video Lectures: MIT 6.868J The Society of Mind (Fall 2011) ]
Machines like Us: Toward AI with Common Sense, Ronald J. Brachman, Hector Levesque (1st Edition), 2022, MIT Press
A Thousand Brains: A New Theory of Intelligence, Jeff Hawkins (2nd Edition), 2022, Basic Books [ Video Lecture ]
The Myth of Artificial Intelligence: Why Computers Can��t Think the Way We Do, Erik J. Larson (1st Edition), 2021, Belknap Press
Genius Makers: The Mavericks Who Brought AI to Google, Facebook, and the World, Cade Metz (1st Edition), 2021, Dutton
What Computers Still Can't Do: A Critique of Artificial Reason, Hubert L. Dreyfus (1st Edition), 1992, MIT Press
A Brief History of Artificial Intelligence: What It Is, Where We Are, and Where We Are Going, Michael Wooldridge (1st Edition), 2021, Flatiron Books
Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence, Kate Crawford (1st Edition), 2021, Yale University Press
Redesigning AI, Daron Acemoglu (1st Edition), 2021, MIT Press
Linguistics for the Age of AI, Marjorie Mcshane, Sergei Nirenburg (1st Edition), 2021, MIT Press
AI Assistants, Roberto Pieraccini (1st Edition), 2021, MIT Press
How Humans Judge Machines, Cesar A. Hidalgo, Diana Orghian, Jordi Albo Canals, Filipa de Almeida, Natalia Martin (1st Edition), 2021, MIT Press
Your Wit Is My Command: Building AIs with a Sense of Humor, Tony Veale (1st Edition), 2021, MIT Press
Python Programming:
Introducing Python for Computer Science and Data Scientists, Paul Deitel, Harvey Deitel (1st Edition), 2020, Pearson
Introduction to Computation and Programming Using Python: With Application to Computational Modeling and Understanding Data, John V. Guttag (3rd Edition), 2021, MIT Press [ Source Code in Python ]
Starting out with Python, Tony Gaddis (5th Edition), 2021, Pearson
Think Python: How to Think Like a Computer Scientist, Allen B. Downey (2nd Edition), 2016, O'Reilly Press (Free Book)
How to Think Like a Computer Scientist: Learning with Python 3, Peter Wentworth, Jeffrey Elkner, Allen B. Downey, Chris Meyers (3rd Edition), 2012 (Free Book)
A Byte of Python, Swaroop C. H. (4th Edition), 2016 (Free Book)
Project Python, Devin Balkcom, 2011 (Free Book)
Python for Everybody: Exploring Data in Python 3, Charles Severance, 2016 (Free Book)
A Hands-On, Project-Based Introduction to Programming, Eric Matthes (2nd Edition), 2016, No Starch Press (Free Book)
Learn Python 3 the Hard Way, Zed A. Shaw (1st Edition), 2017, Addison-Wesley
Introducing Python: Modern Computing in Simple Packages, Bill Lubanovic (2nd Edition), 2019, O'Reilly Press
Clean Code in Python: Develop Maintainable and Efficient Code, Mariano Anaya (2nd Edition), 2021, Packt
Python Programming for Data Science / Analytics:
Python Data Science Handbook: Essential Tools for Working with Data, Jake VanderPlas (1st Edition), 2017, O'Reilly Press
Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython, Wes McKinney (2nd Edition), 2017, O'Reilly Press
Data Science from Scratch: First Principles with Python, Joel Grus (2nd Edition), 2019, O'Reilly Press
Introduction to Machine Learning with Python: A Guide for Data Scientists, Andreas C. Muller, Sarah Guido (1st Edition), 2017, O'Reilly Press
Data Visualization:
Data Visualization: A Practical Introduction, Kieran Healy (1st Edition), 2019, Princeton University Press
Visualization Analysis and Design,Tamara Munzner (1st Edition), 2014, CRC Press
The Visual Display of Quantitative Information, Edward R. Tufte (2nd Edition), 2001, Graphics Press
Fundamentals of Data Visualization - A Primer on Making Informative and Compelling Figures, Claus O. Wilke (1st Edition), 2019, O'Reilly Press (Free Book)
Making Data Visual - A Practical Guide to Using Visualization for Insight, Danyel Fisher, Miriah Meyer (1st Edition), 2018, O'Reilly Press
Google Machine Learning Glossary
An overview of Python Data Visualization libraries
Philip W. L. Fong, 2009. Reading a Computer Science research paper. ACM SIGCSE Bulletin, 41(2), pp.138-140
You and Your Research by Richard Hamming (Bell Labs / NPS). Bell Communications Research Colloquium Seminar, 7 March 1986
An Online LaTeX Editor: Overleaf
LaTeX Tutorial (Overleaf): Learn LaTeX in 30 minutes
The Not So Short Introduction to LaTeX by Tobias Oetiker, Hubert Partl, Irene Hyna, Elisabeth Schlegl, 2021
How To Speak / Present (Video) by Patrick Winston (MIT)