Machine Learning Seminar Series

First-order methods (FOMs) have recently been applied and analyzed for solving problems with complicated functional constraints. Existing works show that FOMs for functional constrained problems have lower-order convergence rates than those for unconstrained problems. In particular, an FOM for a smooth strongly-convex problem can have linear convergence, while it can only converge sublinearly for a constrained problem if the projection onto the constraint set is prohibited. In this talk, I will first give a lower-bound result of FOM for solving affine-constrained problems. Then I will show that the slower convergence is caused by the large number of functional constraints but not the constraints themselves. When there are only O(1) functional constraints, I will show that an FOM can have almost the same convergence rate as that for solving an unconstrained problem, even without the projection onto the feasible set. Finally, I will give an adaptive primal-dual method for problems with many constraints. Experimental results on quadratically-constrained quadratic programs will be shown to demonstrate the theory.

This work describes how machine learning may be used to develop accurate and efficient nonlinear dynamical systems models for complex natural and engineered systems. We explore the sparse identification of nonlinear dynamics (SINDy) algorithm, which identifies a minimal dynamical system model that balances model complexity with accuracy, avoiding overfitting. This approach tends to promote models that are interpretable and generalizable, capturing the essential “physics” of the system. We also discuss the importance of learning effective coordinate systems in which the dynamics may be expected to be sparse. This sparse modeling approach will be demonstrated on a range of challenging modeling problems in fluid dynamics, and we will discuss how to incorporate these models into existing model-based control efforts.

The theoretical and empirical performance of Empirical Risk Minimization (ERM) often suffers when loss functions are poorly behaved with large Lipschitz moduli and spurious sharp minimizers. We propose and analyze a counterpart to ERM called Diametrical Risk Minimization (DRM), which accounts for worst-case empirical risks within neighborhoods in parameter space. DRM has generalization bounds that are independent of Lipschitz moduli for convex as well as nonconvex problems and it can be implemented using a practical algorithm based on stochastic gradient descent. Numerical results illustrate the ability of DRM to find quality solutions with low generalization error in sharp empirical risk landscapes from benchmark neural network classification problems with corrupted labels.

Graphs are powerful mathematical structures to describe relations or interactions among objects in different fields, such as biology, social science, economics and so on. Recent technological innovations are enabling scientists to capture enormous graph-structured data at increasing speed and scale. Thus, a compelling need exists to develop novel learning tools to foster and fuel the next generation of scientific discovery in graph data related research. However, the major computational challenges are due to the unprecedented scale and complexity of complex graph data analytics. There is a critical need for large-scale learning strategies with theoretical guarantees to bridge the gap and facilitate knowledge discovery from complex graph data. This talk will introduce our recent work on developing novel deep graph learning methods to efficiently and effectively process atom graph data for predicting the chemical or biological properties of drug molecules.

Healthcare data is special. Its complex nature is a double-edged sword, possessing great potential but also presenting many difficulties to overcome. For example, administrative claims data —in contrast to the common data types (text, vision, audio) where AI has made eye-popping advances—is multi-modal (i.e., consisting of distinct data types including medical claims, pharmacy claims, and lab results), asynchronous (medication histories and diagnosis histories need not be aligned in time), and irregularly sampled (we only collect data when an individual interacts with the system). Along with such rich and complex data, there is a great deal of domain knowledge in various forms in the healthcare field. In this talk, I will present our work on deep learning architectures for incorporating multimodal data and domain knowledge into models.

Inspired by the proposal of tangent kernels of neural networks (NNs), a recent research line aims to design kernels with a better generalization performance on standard datasets. Indeed, a few recent works showed that certain kernel machines perform as well as NNs on certain datasets, despite their separations in specific cases implied by theoretical results. Furthermore, it was shown that the induced kernels of convolutional neural networks perform much better than any former handcrafted kernels. These empirical results pose a theoretical challenge to understanding the performance gaps in kernel machines and NNs in different scenarios.

In this talk, we show that data structures play an essential role in inducing these performance gaps. We consider a few natural data structures, and study their effects on the performance of these learning methods. Based on a fine-grained high dimensional asymptotics framework of analyzing random features models and kernel machines, we show the following: 1) If the feature vectors are nearly isotropic, kernel methods suffer from the curse of dimensionality, while NNs can overcome it by learning the best low-dimensional representation; 2) If the feature vectors display the same low-dimensional structure as the target function (the spiked covariates model), this curse of dimensionality becomes milder, and the performance gap between kernel methods and NNs become smaller; 3) On datasets that display some invariance structure (e.g., image dataset), there is a quantitative performance gain of using invariant kernels (e.g., convolutional kernels) over inner product kernels. Beyond explaining the performance gaps, these theoretical results can further provide some intuitions towards designing kernel methods with better performance.

abstract

The recent success of machine learning suggests that neural networks may be capable of approximating high-dimensional functions with controllably small errors. As a result, they could outperform standard function interpolation methods that have been the workhorses of current numerical methods. This feat offers exciting prospects for scientific computing, as it may allow us to solve problems in high-dimension once thought intractable. At the same time, looking at the tools of machine learning through the lens of applied mathematics and numerical analysis can give new insights as to why and when neural networks can beat the curse of dimensionality. I will briefly discuss these issues, and present some applications related to solving PDE in large dimensions and sampling high-dimensional probability distributions.

Federated learning (FL) has emerged as a promising approach for distributed machine learning over edge devices, in order to strengthen data privacy, reduce data migration costs, and break regulatory restrictions. A key component of FL is "secure model aggregation", which aims at protecting the privacy of each user’s individual model, while allowing their global aggregation. This problem can be viewed as a privacy-preserving multi-party computing, but with two interesting twists: (1) some users may drop out during the protocol (due to poor connectivity, low battery, unavailability, etc); (2) there is potential for multi-round privacy leakage, even if each round is perfectly secure. In this talk, I will first provide a brief overview of FL, then discuss several recent results on secure model aggregation, and finally end the talk by highlighting a few open problems in the area.

The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. In this talk I will discuss some general mathematical principles allowing for efficient optimization in over-parameterized non-linear systems, a setting that includes deep neural networks. I will discuss that optimization problems corresponding to these systems are not convex, even locally, but instead satisfy the Polyak-Lojasiewicz (PL) condition on most of the parameter space, allowing for efficient optimization by gradient descent or SGD. I will connect the PL condition of these systems to the condition number associated with the tangent kernel and show how a non-linear theory for those systems parallels classical analyses of over-parameterized linear equations. As a separate related developement, I will discuss a perspective on the remarkable recently discovered phenomenon of transition to linearity (constancy of NTK) in certain classes of large neural networks. I will show how this transition to linearity results from the scaling of the Hessian with the size of the network controlled by certain functional norms. Combining these ideas, I will show how the transition to linearity can be used to demonstrate the PL condition and convergence for a general class of wide neural networks. Finally I will comment on systems which are ''almost'' over-parameterized, which appears to be common in practice.

Recent advances in computer vision and artificial intelligence caused a paradigm shift in medical image computing and radiological image analysis. Deep learning has been widely applied to many radiological applications, replacing, or working together with conventional methods. The advantage of being able to learn from that data directly is promising for many imaging tasks. Some key factors and current challenges preventing the widespread adaption of machine learning techniques in the clinic are algorithmic considerations, computational power, and, most critically, high-quality data for training.

NVIDIA wants to provide solutions to make the widespread adoption of deep learning and artificial intelligence easier in the real world. This talk will highlight NVIDIA’s efforts in the healthcare sector and medical imaging research, for example, around federated learning and COVID-19 image analysis, and introduce platforms & hardware considerations for modern machine learning at scale.

Transfer learning has fundamentally changed the landscape of natural language processing (NLP). Many state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. When we only have limited supervision for the downstream tasks, however, due to the extremely high complexity of pre-trained models, aggressive fine-tuning often causes the fine-tuned model to overfit the training data of downstream tasks and fail to generalize to unseen data.

To address such a concern, we propose a new approach for fine-tuning of pretrained models to attain better generalization performance. Our proposed approach adopts three important ingredients: (1) Smoothness-inducing adversarial regularization, which effectively controls the complexity of the massive model; (2) Bregman proximal point optimization, which is an instance of trust-region algorithms and can prevent aggressive updating; (3) Differentiable programming, which can mitigate the undesired bias induced by conventional adversarial training algorithms. Our experiments show that the proposed approach significantly outperforms existing methods in multiple NLP tasks. In addition, our theoretical analysis provides some new insights of adversarial training for improving generalization.

Representations of atomistic systems for machine learning must transform predictably under the geometric transformations of 3D space, in particular rotation, translation, mirrors, and permutation of atoms of the same species. These constraints are typically satisfied by means of atomistic representations that depend on scalar distances and angles, leaving the representation invariant under the above transformations. Invariance, however, limits the expressivity and can lead to an incompleteness of representations. In order to overcome this shortcoming, we recently introduced Neural Equviariant Interatomic Potentials [1], a Graph Neural Network approach for learning interatomic potentials that uses a E(3)-equivariant representation of atomic environments. While most current Graph Neural Network interatomic potentials use invariant convolutions over scalar features, NequIP instead employs equivariant convolutions over geometric tensors (scalar, vectors, …), providing a more information-rich message passing scheme. In my talk, I will first motivate the choice of an equivariant representation for atomistic systems and demonstrate how it allows for the design of interatomic potentials at previously unattainable accuracy. I will discuss applications on a diverse set of molecules and materials, including small organic molecules, water in different phases, a catalytic surface reaction, proteins, glass formation of a lithium phosphate, and Li diffusion in a superionic conductor. I will then show that NequIP can predict structural and kinetic properties from molecular dynamics simulations in excellent agreement with ab-initio simulations. The talk will then discuss the observation of a remarkable sample efficiency in equivariant interatomic potentials which outperform existing neural network potentials with up to 1000x fewer training data and rival or even surpass the sample efficiency of kernel methods. Finally, I will discuss potential reasons for the high sample efficiency of equivariant interatomic potentials.