Machine Learning

Eli Grigsby's whiteboard:

Description:  Every machine learning problem is at its core a constrained optimization problem. One begins with a well-defined prediction task: e.g., predict whether an e-mail is spam, or predict the next word in a sentence. Then

• choose a parameterized function class to predict the labels from the inputs,

• initialize the  parameters somewhere, and 

• take small steps in parameter space to improve the performance of the function on a finite pre-labeled data set known as the training set. 

If one makes good choices, the trained model will generalize well to unseen data drawn from the same probability distribution. 

A neural network is a parameterized function class particularly well-suited for use in ML models. The class of feedforward neural network functions with ReLU activation function is known to be precisely the class of piecewise linear functions with finitely many pieces. 

Tropical geometry also gives a beautiful, coherent way of describing and studying piecewise linear functions between Euclidean spaces, but the two ways of describing PL functions appears to be quite different. 

For example, in Summer 2023, BC undergraduate Jian Huang proved that it is not possible to represent the simple tropical function max{0,x,y} generically (i.e., except in degenerate cases) with ReLU network architecture (2,3,2,1).