Cool Things Made by Others

HERE is pytorch code for an ADAM-type optimizer invented by some of my collages ("The file containing the optimizer is AGNES.py. Everything else is for the specific experiments in the paper.")
HERE is a package that lets you call summary(model) on a pytorch model, returning similarly useful output as the model.summary() method of keras models.
HERE is a GitHub repo that my colleague Tushar Pandey has put together as an introduction to topological data analysis.
HERE is an elegantly simple chatbot (basically, a lookup table) implemented in base python.
HERE the best overall YouTube channel I've found on machine learning: DigitalSreeni.
HERE is a neat 3d visualization of what is happening inside of a small, well-trained CNN made by Adam Harley (thanks, Devon, for sharing this with me!).
HERE is a phenomenal (non-technical) explanation of the fashionable "NPU:" basically just a compute core optimized for a particular pattern of FLOPs.
HERE is a nice tutorial series on pytorch-lightning.

Cool Things that I Made

HERE is my development branch for some open-source software which Los Alamos hired me to write as a summer intern.
HERE is a custom, minimalist training pipeline for pytorch models, which I wrote, inspired by the model.fit() method of keras models.
HERE are some ML demos that I wrote for my advisor's class MATH 680 at Texas A&M.

Notes to Self

I don't like the common practice in model selection (which either happens as part of hyperparameter tuning, or during the incessant "scoreboarding" between different models) of simply choosing the model that gives the smallest validation loss. If a very bloated model gives a val loss of 0.0001 while a much smaller model gives a val loss of 0.0002, then I prefer the latter for its simplicity: it has less memory/compute burden, and it's perhaps less likely to overfit yet unseen data. Instead of choosing the model with the smallest loss, say min_val_loss, I prefer choosing the simplest model with val loss on the same magnitude as min_val_loss, e.g.,, the model with the smallest memory footprint among all those models with val loss of at most 5*min_val_loss.
To improve stability and combat overfitting, try augmenting your training data by pollution with (multiple realizations of) a little Gaussian noise (which, also, results in a larger dataset).
It may be a good idea to preprocess your data by standardizing it (i.e., subtract the empirical mean, then divide by the empirical standard deviation). However, to avoid data leakage, it's best to do so only after the train/test/val split.
I've found that reducing the learning rate may solve "the problem of vanishing gradients" (which, imo, should instead be called, "the problem of prematurely vanish gradients"). I have no idea why this solves the problem. It's actually rather mysterious to me...
Another empirical observation about learning rate: an indication that it is set too large is if the training loss increases after a certain point (in much the same way as one would expect the validation loss to increase after a certain point). When the evolution of a univariate network throughout training is visualized as a .gif, this behavior in the training loss corresponds to the fitted model "bouncing around" as if experiencing turbulence.

How to Get Into ML

Personally, I was introduced to the field by taking a graduate computer science course on the subject, the curriculum of which went something like THIS plus neural networks. If you're a current student, you may want to consider taking such a class at your institution.

While a mathematician may be inclined to read a textbook on the subject (and I can recommend one or two), I suggest focusing on coding, instead. I'd start with the official pytorch tutorial HERE. If you're not that experienced in programming, then you can find a greatly expanded pytorch tutorial HERE. However, you should be aware that your programming ability will be a limiting factor if you're considering a career in ML. So, if you struggle to follow the official pytorch tutorial, you'll need to be prepared to put a lot of effort into improving your programming chops.

For those who insist on reading something like a textbook, here are some starting points which I can recommend (in no order):

Stephan's lecture notes HERE, which include a little bit of code under the designation "Practical aspects."
Chapters 19 and 24 of my advisor's textbook HERE which have more code. Specifically, I propose attempting the exercises in week 6 of the labs HERE, which correspond to chapter 19, as well as the exercises in weeks 8-12, which correspond to chapter 24.