Machine learning models tend to rely on an abundance of training data. Yet, understanding the underlying structure of this data—and models' exact dependence on it---remains a challenge.

In this talk, we will present a framework for directly modeling predictions as functions of training data. This framework, given a dataset and a learning algorithm, pinpoints---at varying levels of granularity---the relationships between train and test point pairs through the lens of the corresponding model class. Even in its most basic version, our framework enables many applications, including discovering data subpopulations, quantifying model brittleness via counterfactuals, and identifying train-test leakage.

Based on joint work with Andrew Ilyas, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry.

There has been a huge wage of excitement for applying Deep learning to medical images. The initial hype has led to many disappointments and painful lessons. But there have been a few successful applications, and the potential for gaining important clinical value from deep learning still exists. The lessons are that one must be very vigilant for subtle biases or unexpected associations that can lead to poor outcomes. But when properly executed, Deep Learning can also lead to valuable insights and improved medical care. This talk will provide a brief coverage of deep learning applied to medical images, and then discuss both the painful lessons, and the great potential of this technology.

Most problems from classical machine learning can be cast as an optimization problem. I will present GENO (GENeric Optimization), a framework that lets the user specify a constrained or unconstrained optimization problem in an easy-to-read modeling language. GENO then generates a solver that can solve this class of optimization problems. The generated solver is usually as fast as hand-written, problem-specific, and well-engineered solvers. Often the solvers generated by GENO are faster by a large margin compared to recently developed solvers that are tailored to a specific problem class. I will dig into some of the algorithmic details, e.g., computing derivatives of matrix and tensor expressions, the optimization methods used in GENO, and their implementation in Python.

Although open-access databases are an important resource in the current deep learning (DL) era, they are sometimes used in an “off label” manner: data published for one task are used during training of algorithms for a different task. In this seminar I will show that this leads to biased, overly optimistic results of well-known inverse problem solvers, focusing on algorithms developed for magnetic resonance imaging (MRI) reconstruction. I will show that when such algorithms are trained using off-label data, they yield biased results, with up to 48% artificial improvement. The underlying cause is that public databases are often preprocessed using hidden pipelines, which change the data features and improve the inverse problem conditioning. My works shows that Compressed Sensing, Dictionary Learning, and Deep Learning algorithms are all prone to this form of bias. Furthermore, once trained, these algorithms exhibit poor generalization to real-world data. To raise awareness to the growing problem of naïve use of public databases, the study refers to publication of biased results as “data crimes”.

Sparsity has given us MP3, JPEG, MPEG, Faster MRI and many fun mathematical problems. Deep generative models like GANs, VAEs, invertible flows and Score-based models are modern data-driven generalizations of sparse structure. We will start by presenting the CSGM framework by Bora et al. to solve inverse problems like denoising, filling missing data, and recovery from linear projections using an unsupervised method that relies on a pre-trained generator. We generalize compressed sensing theory beyond sparsity, extending Restricted Isometries to sets created by deep generative models. Our recent results include establishing theoretical results for Langevin sampling from full-dimensional generative models, generative models for MRI reconstruction and fairness guarantees for inverse problems.

Consider a scenario where we are training a model or performing statistical analyses on a shared dataset with entries collected from several contributing individuals. Differential privacy provides protection against membership inference attacks that try to reveal sensitive information. Robust estimators provide protection against data poisoning attacks where malicious contributors inject corrupted data. Even though both types of attacks are powerful and easy to launch in practice, there is no practical algorithm providing protection against both simultaneously. In the first half of this talk, I will present the first efficient algorithm that guarantees both differential privacy and robustness to the corruption of a fraction of the data. I will focus on the canonical problem of mean estimation, which is a critical building block in many algorithms including stochastic gradient descent for training deep neural networks. In the second half of this talk, I will present a new framework that bridges differential privacy and robust statistics, which we call High-dimensional Propose-Test-Release (HPTR). This is a computationally intractable approach, but universally applicable to several statistical estimation problems including mean estimation, linear regression, covariance estimation, and principal component analysis. In most of these cases, HPTR achieves a near-optimal sample complexity by exploiting robust statistics in the algorithm, thus characterizing the minimax error rate of the corresponding private estimation problems for the first time. This talk is based on two papers: https://arxiv.org/abs/2102.09159 and https://arxiv.org/abs/2111.06578 .

It is well known that image classifiers are vulnerable to adversarial perturbations, and many defenses have been suggested to mitigate this phenomenon. However, testing the effectiveness of a defense is not straightforward. We propose a protocol for standardized and accurate evaluation of a large class of adversarial defenses, which allows to benchmark and track the progress of adversarial robustness in several threat models. Finally, we discuss the current limitations of standardized evaluations, and in which cases adaptive attacks might still be necessary.

The future of machine learning lies in moving both data collection as well as model training to the edge. The emerging area of federated learning seeks to achieve this goal by orchestrating distributed model training using a large number of resource-constrained mobile devices that collect data from their environment. Due to limited communication capabilities as well as privacy concerns, the data collected by these devices cannot be sent to the cloud for centralized processing. Instead, the nodes perform local training updates and only send the resulting model to the cloud. A key aspect that sets federated learning apart from data-center-based distributed training is the inherent heterogeneity in data and local computation at the edge clients. In this talk, I will present our recent work on tackling computational heterogeneity in federated optimization, firstly in terms of heterogeneous local updates made by the edge clients, and secondly in terms of intermittent client availability.

Recent advances in mobile health (mHealth) technology provide an effective way to monitor individuals' health statuses and deliver just-in-time personalized interventions. However, the practical use of mHealth technology raises unique challenges to existing methodologies on learning an optimal dynamic treatment regime. Many mHealth applications involve decision-making with large numbers of intervention options and under an infinite time horizon setting where the number of decision stages diverges to infinity. In addition, temporary medication shortages may cause optimal treatments to be unavailable, while it is unclear what alternatives can be used. To address these challenges, we propose a Proximal Temporal consistency Learning (pT-Learning) framework to estimate an optimal regime that is adaptively adjusted between deterministic and stochastic sparse policy models. The resulting minimax estimator avoids the double sampling issue in the existing algorithms. It can be further simplified and can easily incorporate off-policy data without mismatched distribution corrections. We study theoretical properties of the sparse policy and establish finite-sample bounds on the excess risk and performance error. The proposed method is implemented by our proximalDTR package and is evaluated through extensive simulation studies and the OhioT1DM mHealth dataset.

Quantum entanglement is the physical phenomenon, the medium, and, most importantly, the resources that enable quantum technologies. In this talk, we survey the recent results in estimating the degree of entanglement of the quantum bipartite model over different random state models and entropies. The problem may also be recast into a data mining problem involving symbolic data.

Causal inference and causal learning methods promise improved generalizability and robustness as compared to conventional machine learning approaches by relying on patterns generated by stable and robust causal mechanisms, rather than potentially spurious correlational patterns. However, causal approaches require making crucial assumptions about a system or data-generating process that may be unverifiable in the absence of interventional (experimental) data. We find that eliciting causal assumptions from domain experts and validating or refuting these assumptions are key challenges to the practical application of these methods. This talk describes our research efforts to address these challenges---e.g., by seeking new sources of causal assumptions---as well as experiences with DoWhy, our open-source causal inference library, and its third-party usage.

Not Provided

In recent years, the fundamental limits of distributed/federated learning have been studied under many statistical models, but often in the setting of horizontally partitioning, where data sets share the same feature space but differ in samples. Nevertheless, vertical federated learning, where data sets differ in features, have been in use in finance and medical care. In this talk, we consider a natural distributed nonparametric estimation problem with vertically partitioned datasets. Under a given budget of communication cost or information leakage constraint, we determine the minimax rates for estimating the density at a given point, which reveals that interactive protocols strictly improves over one-way protocols. Our novel estimation scheme in the interactive setting is constructed by carefully identifying a set of auxiliary random variables. The result also implies that interactive protocols strictly improve over one-way for biased binary sequences in the Gap-Hamming problem. (arXiv 2107.00211)

In this talk, we will focus on the emerging field of (adversarially) robust machine learning. The talk will be self-contained and no particular background on robust learning will be needed. Recent progress in this field has been accelerated by the observation that despite unprecedented performance on clean data, modern learning models remain fragile to seemingly innocuous changes such as small, norm-bounded additive perturbations. Moreover, recent work in this field has looked beyond norm-bounded perturbations and has revealed that various other types of distributional shifts in the data can significantly degrade performance. However, in general our understanding of such shifts is in its infancy and several key questions remain unaddressed.

Many supervised learning problems involve high-dimensional data such as images, text, or graphs. In order to make efficient use of data, it is often useful to leverage priors in the problem at hand, such as invariance to certain transformations or stability to small deformations. Empirically, deep convolutional architectures have been very successful on such problems, raising the question of how they are able to capture the structure of these problems for efficient learning. I study this question from a theoretical perspective using kernel methods, in particular convolutional kernels, which are constructed following similar architectural principles, and provide good empirical performance on standard vision benchmarks such as Cifar10. I will present three contributions that highlight the benefits of (deep) convolutional architectures in terms of stability to deformations and sample complexity.

Qizhi He is an Assistant Professor in the Department of Civil, Environmental, and Geo- Engineering at the University of Minnesota. He received his M.A. in applied mathematics and Ph.D. in structural engineering and computational science from UC San Diego, in 2016 and 2018, respectively. Afterwards, he was a Postdoctoral Research Associate at Pacific Northwest National Laboratory (PNNL), where he developed scientific machine learning methods for modeling flow and transport processes in porous media. His current research interests lie at the intersection of computational mechanics, materials modeling, and data-driven computing, with a focus on advancing data-driven machine learning enabled computational tools to predict mechanics of complex multiphysical processes and improve our fundamental understanding of multiscale materials and structures in engineered and natural systems.

Datasets are mathematical objects (e.g., point clouds, matrices, graphs, images, fields/functions) that have shape. This shape encodes important knowledge about the system under study. Topology is an area of mathematics that provides diverse tools to characterize the shape of data objects. In this work, we study a specific tool known as the Euler characteristic (EC). The EC is a general, low-dimensional, and interpretable descriptor of topological spaces defined by data objects. We revise the mathematical foundations of the EC and highlight its connections with statistics, linear algebra, field theory, and graph theory. We discuss advantages offered by the use of the EC in the characterization of complex datasets; to do so, we illustrate its use in different applications of interest in chemical engineering such as process monitoring, flow cytometry, and microscopy. We show that the EC provides a descriptor that effectively reduces complex datasets and that this reduction facilitates tasks such as visualization, regression, classification, and clustering.