The Grand Vision:

I might call my thesis something like "In Search of a Unifying Theory (and Practice) of Non-Linear Approximation." Many methods fall under “non-linear approximation.” Some from statistics and signal processing, like compressive sensing, low-rank matrix recovery, and "super-resolution," are well understood. Meanwhile, others from machine learning like kernel approximation, Gaussian mixture models, and shallow neural networks are less understood and even more interesting.

Researchers in these areas (aside from applied machine learning engineers, who don't care) are mostly aware of the fact that these methods are intuitively analogous. However, theoretical results linking them in a concrete manner are mysteriously absent. My work aims to build a theoretical framework which is a sufficiently general that results -- and, more importantly, tools -- from the well-understood cases can be translated to create new results/tools in the less understood cases, where such advancements are sorely needed. Essentially, we're trying to redesign neural networks from square one, borrowing ideas from signal processing to reformulate them in a manner that they'll be more well-behaved.

If successful, this would result in the first ever theoretical approach to training a neural network, in contrast to the contemporary reliance exclusively on empirical methods. This is important, not only because models are more trustworthy when supported theoretically, but also because empirical methods are a variable cost (each model you want to fit needs its hyper-parameters tuned) whereas theory-building is a fixed cost, which is more efficient in the long run (e.g., if you could come up with a formula for the "optimal" learning rate, then no one would need to tune it ever again, resulting in a lot of time and money saved as years go on). In fact, we have already solved this problem for the very simplest ReLU networks (unpublished).

An Example with Neural Networks:

In order to train a neural network (i.e., fit a neural network to some data), you have to "solve" a math problem, which can be simply explained as minimizing the loss function. It's bad enough that this a difficult problem to solve (because the loss function is non-convex) but, worse yet, it's the wrong problem entirely! Indeed, most architectures are capable of achieving a "training loss" of zero. However, this would almost certainly entail overfitting. Therefore, one secretly hopes to not exactly minimize the loss function, relying on a validation dataset to assess whether or not the loss has gotten "too small."

Suffice it to say, the whole thing is just highly ill-posed from a mathematical viewpoint. In practice, this ill-posedness means there isn't a reliable recipe for training an AI: it's not as concrete as solving a math problem. Instead, experts with years of experience in hyperparameter tuning and whatnot need to convince the problem to not exactly solve itself in precisely the right manner for the neural network to actually work. Because of this, machine learning sometimes feels like more of an art than a science. The cost of development is high, due to the necessity of expert guidance, and there can always be unexpected failure cases, due to the mathematical ill-posedness.

My advisor has proposed a more difficult math problem than the one currently in use. Loosely speaking, the problem is more difficult because it is well-posed: cutting corners isn't allowed. This would result in more trustworthy, reliable AI if you solved it instead of the conventional problem. The issue is... it's a really hard math problem! If I could solve the problem for shallow ReLU networks, that would be my PhD thesis. For context, shallow ReLU networks are, arguably, the simplest neural networks that are powerful enough for basic science/enterprise purposes (recall: they're "powerful enough" to be universal approximators).