FUTURE TALKS

May 22: Søren Hauberg (Technical University of Denmark))

TBA

May 29: Mark Transtrum (Brigham Young University)

eGAD! Double descent is Explained by the Generalized Aliasing Decomposition

A central problem in data science is to use potentially noisy samples of an unknown function to predict values for unseen inputs. Classically, predictive error is understood as a trade-off between bias and variance that balances model simplicity with its ability to fit complex functions. However, over-parameterized models exhibit counterintuitive behaviors, such as “double descent” in which models of increasing complexity exhibit decreasing generalization error. Other models may exhibit more complicated patterns of predictive error with multiple peaks and valleys. Neither double descent nor multiple descent phenomena are well explained by the bias–variance decomposition. I present the generalized aliasing decomposition (GAD) to explain the relationship between predictive performance and model complexity. The GAD decomposes the predictive error into three parts: 1.) model insufficiency, which dominates when the number of parameters is much smaller than the number of data points, 2.) data insufficiency, which dominates when the number of parameters is much greater than the number of data points, and 3.) generalized aliasing, which dominates between these two extremes. I apply the GAD to linear regression problems from machine learning and materials discovery to explain salient features of the generalization curves in the context of the data and model class.