Transport dominated dynamics are challenging for data 

assimilation because solutions have often low regularity, they are 

spatially localized, and the evolution preserves certain quantities. In 

this talk, I will present recovery algorithms to approximate two classes 

of such transport problems from partial observations: Hamiltonian 

dynamics and conservation laws. The main ingredients of the algorithms 

are structure preserving reduced order models, and a dynamical strategy 

to position sensors.

We introduce a new methodology 'charcoal' for estimating the location of sparse changes in high-dimensional linear regression coefficients, without assuming that those coefficients are individually sparse. The procedure works by constructing different sketches (projections) of the design matrix at each time point, where consecutive projection matrices differ in sign in exactly one column. The sequence of sketched design matrices is then compared against a single sketched response vector to form a sequence of test statistics whose behavior shows a surprising link to the well-known CUSUM statistics of univariate changepoint analysis. The procedure is computationally attractive, and strong theoretical guarantees are derived for its estimation accuracy. Simulations confirm that our methods perform well in extensive settings, and a real-world application to a large single-cell RNA sequencing dataset showcases the practical relevance.

Partial differential equations (PDEs) are among the most universal tools used in modeling problems in nature and man-made complex systems. Nearly all traditional approximation algorithms for PDEs in the literature suffer from the so-called "curse of dimensionality" in the sense that the number of required computational operations of the approximation algorithm to achieve a given approximation accuracy grows exponentially in the dimension of the considered PDE. With such algorithms it is impossible to approximately compute solutions of high-dimensional PDEs even when the fastest currently available computers are used. In the specific situation of linear parabolic PDEs and approximations at a fixed space-time point, the curse of dimensionality of deterministic methods can be overcome by means of Monte Carlo approximation algorithms and the Feynman-Kac formula. In this talk we show that deep neural networks (DNNs) have the fundamental property to be able to approximate solutions of semilinear PDEs with Lipschitz nonlinearities with the number of real parameters of the approximating DNN growing at most polynomially in, both, the reciprocal of the prescribed approximation accuracy and the PDE dimension. Our arguments are strongly based on suitable nonlinear Monte Carlo methods for such PDEs. In the second part of the talk we present and analyze acceleration techniques for stochastic gradient descent optimization in the context of deep learning approximations for PDEs and optimal control problems.

The talk starts with a brief introduction into machine learning applications utilizing neural network representations. Although this is a rapidly advancing field, many aspects remain mathematically unresolved. I will outline the standard algorithms currently employed for training neural networks, focusing on one very effective and popular method in practical applications: the Adam algorithm, introduced by Kingma and Ba in 2014. This algorithm is an enhancement of the traditional stochastic gradient descent, incorporating a momentum approach together with a damping factor. During the presentation, I will present new error estimates for the Adam algorithm, developed with Arnulf Jentzen. Specifically, we demonstrate that the algorithm’s effective behavior is closely connected to a particular vector field, that we call the Adam field.

For several classes of neural PDE solvers (Deep Ritz, PINNs, DeepONets), the ability to approximate the solution or solution operator to a partial differential equation (PDE) hinges on the ability of a neural network to approximate the solution in the spatial variables. We analyze the capacity of neural networks to approximate solutions to an elliptic PDE assuming that the boundary condition can be approximated efficiently. Our focus is on the Laplace operator with Dirichlet boundary conditions in Barron space on the unit square. We show that solutions generally are not Barron functions, but can be approximated very efficiently by functions of low Barron norm in several Sobolev norms. This is joint work with Malhar Vaishampayan.

In this talk, I will present recent progress on a geometric perspective of the benign overfitting phenomenon for the minimum norm interpolant estimator in linear regression with respect to squared excess risk. By analyzing the self-regularization property of the minimum norm interpolant estimator, we derive sufficient conditions for benign overfitting. Specifically, for the minimum ℓ2\ell_2-norm interpolant estimator, we establish necessary and sufficient conditions for benign overfitting under the L4+ϵL_{4+\epsilon}-L2L_2 norm equivalence assumption. For the minimum ℓ1\ell_1-norm interpolant estimator, we obtain sufficient conditions for benign overfitting in the extremal high-dimensional regime, for arbitrary signals with anisotropic covariance.

Our approach is purely geometric: using the Dvoretzky-Milman theorem, we establish the self-regularization property of the minimum norm interpolant estimator. We prove that this estimator automatically induces an optimal decomposition of the feature space, such that its projection onto a certain subspace corresponds to a regularized empirical risk minimizer, where the regularization parameter is automatically determined by the Dvoretzky-Milman dimension of the complementary subspace. If time permits, I will also discuss sufficient conditions for benign overfitting in the case of the minimum ℓq\ell_q-norm interpolant estimator, as well as its applications to benign overfitting in classification problems.

We consider the problem of learning regularizers for linear discrete

inverse problems. We focus on the supervised setting and on variational

models. Our goal is to understand when and how this is possible. To that

end, we do a theoretical analysis of affine linear regularization

methods that are derived from quadratic Tikhonov regularization (we call

the quadratic regularization and Lavrentiev regularization). This

naturally leads to bilevel problems which we can turn into nonlinearly

constrained optimization problems. We will identify situations in which

Tikhonov does give good reconstruction quality in the least squares

sense. On finding is: If one includes learning of the discrepancy term

(i.e. we learn the noise covariance), all methods perform equal and best

possible. We also show that if one does not learn the discrepancy term,

Tikhonov regularization is worse than the other methods, depending on

the interplay of the noise and the operator.

This is joint work with Sebastian Banert, Christoph Brauer and Lionel Tondji.

We study the advantages of using accelerated gradient methods, specifically based on the Frank-Wolfe method and projected gradient descent, for privacy and heavy-tailed robustness. Our approaches are as follows: For the Frank-Wolfe method, our technique is based on a tailored learning rate and a uniform lower bound on the gradient l_2-norm over the constraint set. For accelerating projected gradient descent, we use the popular variant based on Nesterov's momentum, and we optimize our objective over R^p. These accelerations reduce iteration complexity, translating into stronger statistical guarantees for empirical and population risk minimization, for instance. Our analysis covers three settings: non-random data, random model-free data, and parametric models (linear regression and generalized linear models). Methodologically, we approach both privacy and robustness based on noisy gradients. We ensure differential privacy via the Gaussian mechanism and advanced composition, and we achieve heavy-tailed robustness using a geometric median-of-means estimator, which also sharpens the dependency on the covariates' dimension. Finally, we compare our rates to existing bounds and identify scenarios where our methods attain optimal convergence.

Efficient and credible multi-query, real-time simulations constitute a critical enabling factor for digital twinning, and data-driven reduced-order modeling is a natural choice for achieving this goal. This talk will discuss two probabilistic methods for the learning of reduced-order dynamics, in which a significantly reduced dimensionality of dynamical systems guarantees improved efficiency, and the endowed uncertainty quantification certifies computational credibility.

The first method is the Bayesian reduced-order operator inference, a non-intrusive approach that inherits the formulation structure of projection-based reduced-state governing equations yet without requiring access to the full-order solvers. The reduced-order operators are estimated using Bayesian inference with Gaussian priors, and two fundamentally different strategies of likelihood definition will be discussed. Recovered as posterior Gaussian distributions conditioning on projected state data, the reduced-order operators probabilistically describe a low-dimensional dynamical system for the predominant latent states, and provide a naturally embedded Tikhonov regularization together with a quantification of modeling uncertainties.

The second method employs deep kernel learning — a probabilistic deep learning tool that integrates neural networks into manifold Gaussian processes — for the data-driven discovery of low-dimensional latent dynamics from high-dimensional measurements given by noise-corrupted images. This tool is utilized for both the nonlinear dimensionality reduction and the representation of reduced-order dynamics. Numerical results have shown the effectiveness of deep kernel learning in the denoising and uncertainty quantification throughout model reduction. 

We consider the problem of learning a linear operator θ between two Hilbert spaces from empirical observations, which we interpret as least squares regression in infinite dimensions. We show that this goal can be reformulated as an inverse problem for θ with the undesirable feature that its forward operator is generally non-compact (even if θ is assumed to be compact or of p-Schatten class). However, we prove that, in terms of spectral properties and regularisation theory, this inverse problem is equivalent to the known compact inverse problem associated with scalar response regression. Our framework allows for the elegant derivation of dimension-free rates for generic learning algorithms under Hölder-type source conditions. The proofs rely on the combination of techniques from kernel regression with recent results on concentration of measure for sub-exponential Hilbertian random variables. The obtained rates hold for a variety of practically-relevant scenarios in functional regression as well as nonlinear regression with operator-valued kernels and match those of classical kernel regression with scalar response. 

In this talk, our aim is to have a unified treatment of the problem of estimating a regression function in one-parameter exponential families. Besides, we want to go beyond the common assumption that the true distribution of the data exactly belongs to the statistical model we consider.  Our estimation methodology is based on Rho-estimation. Strategies based on a single model and model selection will be discussed, where each model is assumed to be a VC-subgraph class. We present non-asymptotic risk bounds for the resulting estimators and explain the robustness with respect to data contamination, the presence of outliers and model misspecification. To remedy the curse of dimensionality, we handle estimation under structure assumptions on the regression functions. We consider specific models in these cases and derive VC dimensional bounds for them. Combining the existing approximation results, we show that under a suitable parametrization of the exponential family, the rates of convergence we get coincide with those derived in the Gaussian regression setting under the same structure assumption. At the end of the talk, we carry out a simulation study to compare the performance of Rho-estimators to the maximum likelihood estimator and median-based ones. 

We propose an ensemble learning algorithm named bagged k-distance for mode-based clustering (BDMBC) by putting forward a new measurement called the probability of localized level sets (PLLS), which enables us to find all clusters for varying densities with a global threshold. On the theoretical side, we show that with a properly chosen number of nearest neighbors k_D in the bagged k-distance, the sub-sample size s, the bagging rounds B, and the number of nearest neighbors k_L for the localized level sets, BDMBC can achieve optimal convergence rates for mode estimation. It turns out that with a relatively small B, the sub-sample size s can be much smaller than the number of training data n at each bagging round, and the number of nearest neighbors k_D can be reduced simultaneously. Moreover, we establish optimal convergence results for the level set estimation of the PLLS in terms of Hausdorff distance, which reveals that BDMBC can find localized level sets for varying densities and thus enjoys local adaptivity. On the practical side, we conduct numerical experiments to empirically verify the effectiveness of BDMBC for mode estimation and level set estimation, which demonstrates the promising accuracy and efficiency of our proposed algorithm.

Compared to artificial neural networks (ANNs), the brain learns faster, generalizes better to new situations and consumes much less energy. ANNs are motivated by the functioning of the brain but differ in several crucial aspects. In particular, it is biologically implausible that the learning of the brain is based on gradient descent. In this talk we look at the brain as a statistical method for supervised learning. The main contribution is to relate the local updating rule of the connection parameters in biological neural networks (BNNs) to a zero-order optimization method. 

The talk is based on arxiv:2301.11777.

Variational Autoencoders are unsupervised machine learning algorithms, that encode data points to points in a so-called latent space. It is often hoped that the encoding carries meaningful information about the datasets, which is sometimes formulated as that it disentangles latent factors. We have introduced the Diffusion Variational Autoencoder to facilitate this disentanglement, in case the latent space has the structure of a Riemannian manifold. An important element of the algorithm is the efficient approximation to the entropy of the heat kernel on a Riemannian manifold. We will discuss recent work in which we derived general asymptotic expansions for this entropy. This is joint work with Luis Pérez Rey, Vlado Menkovski and Mahefa Ravelonanosy.

Laplacian eigenmaps and diffusion maps are nonlinear dimensionality reduction methods that use the eigenvalues and eigenvectors of normalized graph Laplacians. From a mathematical perspective, the main problem is to understand these empirical Laplacians as spectral approximations of the underlying Laplace-Beltrami operator. In this talk, we study Laplacian eigenmaps through the lens of kernel PCA. This leads to novel points of view and allows to leverage results for empirical covariance operators in infinite dimensions.

We propose an ensemble learning algorithm called medians of forests for robust density estimation (MFRDE), which achieves robustness against outliers through a pointwise median operation on forest density estimators fitted on subsampled datasets. Compared to robust kernel-based methods, the local property of MFRDE enables us to choose larger subsampling sizes, sacrificing less accuracy for density estimation while achieving robustness. On the theoretical side, by introducing new concepts on local outliers, we show that even if the number of outliers reaches a certain polynomial order in the sample size, MFRDE is able to achieve almost the same convergence rate as the same algorithm on uncontaminated data, whereas robust kernel-based methods fail. On the practical side, real data experiments show that MFRDE outperforms existing robust kernel-based methods. Moreover, we apply MFRDE to anomaly detection to showcase a further application.

In this talk we discuss existence of global minima in optimisation problems over shallow neural networks. More explicitly, the function class over which we minimise is the family of all functions that can be expressed as artificial residual or feedforward neural networks with one hidden layer featuring a specified number of neurons with ReLU (or Leaky ReLU) activation. We give existence results. Moreover, we provide counterexamples that illustrate the relevance of the assumptions imposed in the theorems.

Computational fluid dynamics is a common tool in cardiovascular science and engineering to simulate, predict and study hemodynamics in arteries. Owing to the complexity and scale of cardiovascular flow problems, especially for patient-specific cases, the model evaluation could be computationally expensive. The domains for these hemodynamic simulations are either generated synthetically based on a simplification of the anatomical geometry or directly segmented from clinical image data. These anatomical geometries of a kind, such as the segments of arteries and organs are similar in terms of general shape but different in details. In this work, a data-driven surrogate model is proposed for the efficient prediction of blood flow simulations on similar but distinct domains. The proposed surrogate model leverages surface registration to parameterise those similar but distinct shapes and formulate corresponding hemodynamics information into geometry-informed snapshots by the diffeomorphism constructed between the reference domain and target domain. A non-intrusive reduced-order model for geometrical parameters is subsequently constructed using proper orthogonal decomposition, and a radial basis function interpolator is trained for predicting the reduced coefficients of the reduced-order model based on reduced coefficients of geometrical parameters of the shape. Two examples of blood flowing through a stenosis and a bifurcation are presented and analysed. The proposed surrogate model demonstrates its accuracy and efficiency in hemodynamics prediction and shows its potential application toward real-time simulation or many-query scenarios.

We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for expectations and covariance matrices of the iterates are derived. In contrast with the widely cited connection between dropout and \ell_2-regularization in expectation, the results indicate a much more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout.

This talk is based on joint work with Sophie Langer and Johannes Schmidt-Hieber

Increasingly high-dimensional data sets require that estimation methods do not only satisfy statistical guarantees but also remain computationally feasible. In modern statistics and machine learning early stopping of algorithms has been identified as one of the tools to address such issues. In this context, we present recent results both on the implicit regularization effect of early stopping and the potential of data driven early stopping rules to perform adaptive estimation in a sequential, computationally efficient manner.

Selecting the best regularization parameter in inverse problems is a classical and yet challenging problem. In the past years, data-driven techniques have gained popularity because, while circumventing some limitations from classical approaches, they seem to perform better in practice. In this talk, we propose and study a statistical learning approach based on empirical risk minimization. Our main contribution is a theoretical analysis, showing that, provided with enough data, this approach can reach sharp rates. Our analysis draws ideas and techniques from statistical machine learning and regularization theory for ill-posed inverse problems. Finally, we present some numerical simulations that corroborate and illustrate the theoretical findings.  

TBA

In this talk we will look at the complexity of computing mixed strategy equilibria for zero-sum matrix games. We will study the query complexity, asking how much of the matrix has to be read to output an approximate mixed equilibrium? We will review the standard first-order query model, and discuss the upper bounds by Rakhlin and Sridharan (2013).  We will then turn to new lower bounds. We will talk about 

* How this question is delicate (and many lower bound techniques cannot not apply), by demonstrating that one query suffices when the entries of the payoff matrix are guaranteed to be rationals. 

* A lower bound showing that for exact equilibria the entire matrix needs to be read (this takes K queries) 

* New techniques giving the first (yet weak) lower bounds in the approximate case. 

No prior knowledge about games or optimization will be assumed. All are welcome! 

Neural networks define functions by composing linear and nonlinear maps in a multi-layer architecture. The functional-analytic study of the function spaces defined by neural networks is a line of research that has recently attracted a lot of attention. Studying the spaces of such functions can provide a new perspective to understand the corresponding learning models and their inductive bias.  We show that neural networks define suitable reproducing kernel Banach spaces. These spaces are equipped with norms that enforce a form of sparsity, enabling them to adapt to potential latent structures within the input data and their representations. In particular, leveraging the theory of reproducing kernel Banach spaces, combined with variational results, we derive representer theorems that justify the finite architectures commonly employed in applications.

In 2017, it was observed that Residual Neural Networks (ResNets) can be

studied as discretization of continuous-time control systems, which

are often called NeurODEs. In the last years, Control Theory has been

fruitfully applied to study the properties of existing networks, and to

develop new ones. Since the dimension of the phase-space of a NeurODE is

constant, they have not been used so far to model Deep Learning

architectures where the dimensions of the inputs and the outputs vary

along the layers. In particular, this is the case for Autoencoders,

where the dimension of the data is compressed during the encoding phase,

and it increases during the decoding. In our work, we model a

continuous-time Autoencoder, which we call AutoencODE, and we extend to

this case the mean-field control framework already developed for

classical NeurODEs. Moreover, we tackle the case of low Tikhonov

regularization, resulting in possibly non-convex landscapes of the cost

functional. It turns out that most of the results holding globally in

case of high Tikhonov regularization can be recovered in regions where the loss is locally convex.

When training neural networks with gradient methods using a small initialisation of the weights, strange types of learning curves appear: the training process makes very little progress for some time, followed by a sharp transition where a new “feature” is suddenly learnt. This behaviour is usually referred to as incremental learning. In this talk, I will show that we can fully describe this phenomenon when considering a toy network architecture. In this simplified setting, we can prove that the gradient flow trajectory jumps from a saddle of the training loss to another. Each visited saddle as well as the jump times can be computed through a recursive algorithm reminiscent of the Homotopy algorithm used for finding the Lasso path.

Recent advances in decentralized multiagent reinforcement learning (MARL) have led to the development of algorithms that are provably convergent in a variety of Markov game subclasses. One of these is the Decentralized Q-learning (DQ) algorithm by Arslan and Yüksel (2017) which is provably convergent in weakly acyclic games. In this talk, I will present a new characterization of weak acyclicity and use it to show that the prisoner's dilemma with a memory of one period is weakly acyclic. This new characterization naturally leads to an identification of the basins of attraction of all possible strategy equilibria of the DQ algorithm. Since only a subset of strategy equilibria leads to robust collusion, we can use this to quantify the likelihood of observing algorithmic collusion. Time permitting, I will discuss the effect that fluctuations in the learning process and the addition of a third intermediate action to the prisoner's dilemma have on the likelihood of collusion. 

The last decade there has been much attention in the media to the fact that many scientific results are not reproducible, especially in medicine and psychology this is widely acknowledged. Part of the problem is due to the mathematics used for hypothesis testing. The standard methodology is the "p-value based null hypothesis significance testing", despite a myriad of problems surrounding it. We present the E-value, a notion of evidence which overcomes some of the issues. On January 24 we presented our paper "Safe Testing" at the discussion meeting of the Royal Statistical Society in London.

Reinforcement learning (RL) and stochastic control share the common goal of finding optimal strategies in uncertain environments. RL, which is a subfield of machine learning, has gained popularity in recent years due to its ability to learn optimal behaviours through trial and error. While RL algorithms are actively used in a wide range of domains, formulating theoretical guarantees and interpretable models is a major challenge. In contrast, the mathematical branch of stochastic control provides theoretical solutions to optimal control problems in many scenarios, but their practicability suffers from the standard assumption of known dynamics of the underlying stochastic process.

To overcome this limitation, we propose purely data-driven strategies for stochastic control, which we investigate for ergodic impulse and singular control problems in the context of continuous diffusion processes. In particular, we describe the specific statistical challenges arising in the stochastic control set-up. The exploration vs. exploitation dilemma, which is familiar from RL, plays an essential role in the considerations, and we present some concentration results allowing to deal with it. Finally, we show how these insights can be translated into regret convergence rates of polynomial order for the considered control problems.

Many problems in imaging and data science require to reconstruct, from partial observations, highly concentrated signals, such as pointwise sources or contour lines. This work introduces a novel algorithm for recovering measures supported on such structured domains, given a finite number of their moments. Our approach is based on the traditional singular value decomposition methodology of subspace methods, but lifts their restriction to the framework of Dirac masses, and is able to recover geometrically faithful discrete approximations of measures with density. The crucial step consists in the approximate joint diagonalization of a few non-commuting matrices, which we perform using a quasi-Newton algorithm. Experiments show that our method performs well, not only in the setting of well separated Dirac masses, as predicted by the standard theory of the truncated moment problem, but also in the case of continuous measures, which is not covered by theoretical guarantees and where usual methods empirically fail. We illustrate its applicability in optimal transport problems, where the coupling measure is often localized on the graph of some function.

Given a sample of covariate-response pairs, we consider the

subgroup selection problem of identifying a subset of the covariate domain

where the regression function exceeds a pre-determined threshold. We

introduce a computationally-feasible approach for subgroup selection in

the context of multivariate isotonic regression based on martingale tests

and multiple testing procedures for logically-structured hypotheses. Our

proposed procedure satisfies a non-asymptotic, uniform Type I error rate

guarantee with power that attains the minimax optimal rate up to

poly-logarithmic factors. Extensions cover classification, isotonic

quantile regression and heterogeneous treatment effect settings.

In various domains, statistical algorithms trained on personal data take pivotal decisions which influence our lives on a daily basis. Recent studies show that a naive use of these algorithms in sensitive domains may lead to unfair and discriminating decisions, often inheriting or even amplifying biases present in data. In the first part of the talk, I will introduce and discuss the question of fairness in machine learning through concrete examples of biases coming from the data and/or from the algorithms. In a second part, I will demonstrate how statistical learning theory can help us better understand and overcome some of those biases. In particular, I will present a selection of recent results from two of my papers on the Demographic Parity constraint:

- A minimax framework for quantifying risk-fairness trade-off in regression (with E. Chzhen), Ann. Statist. 50(4): 2416-2442 (Aug. 2022). DOI: 10.1214/22-AOS2198;

- Fair learning with Wasserstein barycenters for non-decomposable performance measures (with S. Gaucher and E. Chzhen), AISTATS 2023.

Random Forests (RFs) are at the cutting edge of supervised machine learning in terms of prediction performance, especially in genomics. Iterative RFs (iRFs) use a tree ensemble from iteratively modified RFs to obtain predictive and stable nonlinear or Boolean interactions of features. They have shown great promise for Boolean biological interaction discovery that is central to advancing functional genomics and precision medicine. However, theoretical studies into how tree-based methods discover Boolean feature interactions are missing. Inspired by the thresholding behavior in many biological processes, we first introduce a discontinuous nonlinear regression model, called the “Locally Spiky Sparse” (LSS) model. Specifically, the LSS model assumes that the regression function is a linear combination of piecewise constant Boolean interaction terms. Given an RF tree ensemble, we define a quantity called “Depth-Weighted Prevalence” (DWP) for a set of signed features S. Intuitively speaking, DWP(S) measures how frequently features in S appear together in an RF tree ensemble. We prove that, with high probability, DWP(S) attains a universal upper bound that does not involve any model coefficients, if and only if S corresponds to a union of Boolean interactions under the LSS model. Consequentially, we show that a theoretically tractable version of the iRF procedure, called LSSFind, yields consistent interaction discovery under the LSS model as the sample size goes to infinity. Finally, simulation results show that LSSFind recovers the interactions under the LSS model, even when some assumptions are violated.

Reference: https://www.pnas.org/doi/10.1073/pnas.2118636119

Co-authors: Yu Wang, Xiao Li, and Bin Yu (UC Berkeley)

Federated learning is gaining increasing popularity, with data heterogeneity and privacy being two prominent challenges. In this paper, we address both issues within a federated transfer learning framework, aiming to enhance learning on a target data set by leveraging information from multiple heterogeneous source data sets while adhering to privacy constraints. We rigorously formulate the notion of federated differential privacy, which offers privacy guarantee for each data set without assuming a trusted central server.  Under this privacy constraint, we study three classical statistical problems, including univariate mean estimation, low-dimensional linear regression, and high-dimensional linear regression. By investigating the minimax rates and identifying the costs of privacy for these problems, we show that federated differential privacy is an intermediate privacy model between the well established local and central models of differential privacy. Our analyses incorporate data heterogeneity and privacy, highlighting the fundamental costs of both in federated learning and underscoring the benefit of knowledge transfer across data sets.  This is joint work with Mengchu Li (Warwick), Ye Tian (Columbia) and Yang Feng (NYU).

The roto-translation group SE(2) has been of active interest in image analysis due to methods that lift the image data to multi-orientation representations defined on this Lie group. This has led to impactful applications of crossing-preserving flows for image de-noising, geodesic tracking, and roto-translation equivariant deep learning.

In this talk, I will enumerate a computational framework for optimal transportation over Lie groups, with a special focus on SE(2). I will describe several theoretical aspects such as the non-optimality of group actions as transport maps, invariance and equivariance of optimal transport, and the quality of the entropic-regularized optimal transport plan using geodesic distance approximations.

Finally, I will illustrate a Sinkhorn like algorithm that can be efficiently implemented using fast and accurate distance approximations of the Lie group and GPU-friendly group convolutions. We report valuable advancements in the experiments on 1) image barycenters, 2) interpolation of planar orientation fields, and 3) Wasserstein gradient flows on SE(2). We observe that our framework of lifting images to SE(2) and optimal transport with left-invariant anisotropic metrics leads to equivariant transport along dominant contours and salient line structures in the image and leads to meaningful interpolations compared to their counterparts on R^2.

*Joint work with Daan Bon, Gijs Bellaard, Olga Mula and Remco Duits from CASA – TU/e. Preprint: https://arxiv.org/abs/2402.15322

We propose an inference method for detecting multiple change points in high-dimensional time series, targeting dense or spatially clustered signals. Our method aggregates moving sum (MOSUM) statistics cross-sectionally by an L^2-norm and maximizes them over time. We further introduce a novel Two-Way MOSUM, which utilizes spatial-temporal moving regions to search for breaks, with the added advantage of enhancing testing power when breaks occur in only a few groups. The limiting distribution of an L^2-aggregated statistic is established for testing break existence by extending a high-dimensional Gaussian approximation theorem to spatial-temporal non-stationary processes. Simulation studies exhibit promising performance of our test in detecting non-sparse weak signals. Two applications on equity returns and COVID-19 cases in the United States show the real-world relevance of our algorithms. The R package “L2hdchange” is available on CRAN.

The theory of optimal transport (OT) offers versatile tools for the comparison of probability measures in a geometrically faithful way. In statistical contexts, transport based methodology often relies on estimation of the OT cost through an empirical plug-in approach, which raises questions about its accuracy. The convergence behavior of the empirical OT cost for increasing sample size is dictated by various aspects. These include the intrinsic dimension of the population measures, their concentration, as well as the regularity of the ground cost function. Remarkably, under distinct population measures with different intrinsic dimensions, the convergence rate for the empirical OT cost adapts to the population measures in the most favorable way, being determined by the lower dimensional measure. This phenomenon represents a hallmark feature of empirical optimal transport and is termed "lower complexity adaptation“. The talk is based on joint work with Thomas Staudt and Axel Munk.

Using nonlinear projections and preserving structure in model order reduction (MOR) are currently active research fields. In this paper, we provide a novel differential geometric framework for model reduction on smooth manifolds, which emphasizes the geometric nature of the objects involved. The crucial ingredient is the construction of an embedding for the low-dimensional submanifold and a compatible reduction map, for which we discuss several options. Our general framework allows capturing and generalizing several existing MOR techniques, such as structure preservation for Lagrangian- or Hamiltonian dynamics, and using nonlinear projections that are, for instance, relevant in transport-dominated problems. The joint abstraction can be used to derive shared theoretical properties for different methods, such as an exact reproduction result. To connect our framework to existing work in the field, we demonstrate that various techniques for data-driven construction of nonlinear projections can be included in our framework.

Many problems in science, like particle physics, electrodynamics, medical imaging, protein engineering, etc., stay unchanged under transformations of the underlying space or spacetime. Deep neural networks that process data from those fields could benefit in terms of data efficiency, parameter complexity and generalization capabilities, if they already incorporated such space (time) symmetries from the start. In this talk we show how this can be achieved for big classes of neural network architectures, like variants of multilayer perceptrons, message passing and convolutional neural networks, etc.

Gromov-Wasserstein optimal transport is a generalization of the classical optimal transport problem and allows the comparison of two arbitrary metric measure spaces via the so-called Gromov-Wasserstein metric.  Due to its invariance under measure- and distance-preserving transformations, this metric has many applications in graph and shape analysis.  Unfortunately, the computation of the Gromov-Wasserstein distance is numerically expensive, limiting its application in machine learning like in classification tasks.  To overcome this issue, in the first part of the talk, we propose a linear version of the Gromov-Wasserstein metric, which is based on the geometric structure of the Gromov-Wasserstein space.  Numerical examples illustrate that the linear Gromov-Wasserstein transport can replace the expensive computation of pairwise Gromov-Wasserstein distances. In the second part, we focus on the computation of Gromov-Wasserstein barycenters, which we characterize as solutions of a multi-marginal formulation. In order to tackle the problem numerically,we adapt the well-known Fréchet mean procedure on Riemannian manifolds to the Gromov-Wasserstein geometry. Finally, we demonstrate the proposed method for 3d shape interpolations.

We study the limiting dynamics of noisy gradient descent systems in the overparameterized regime. In this regime the set of global minimizers of the loss is large, and when initialized in a neighbourhood of this zero-loss set a noisy gradient descent algorithm slowly evolves along this set. In some cases this slow evolution has been related to better generalisation properties. We give an explicit characterization of this evolution for the broad class of noisy gradient descent systems. Our results show that the structure of the noise affects not just the form of the limiting process, but also the time scale at which the evolution takes place. We apply our theory to Dropout, label noise and classical SGD (minibatching) noise. We show that dropout and label noise models evolve on two different time scales. At the same time classical SGD yields a trivial evolution on both mentioned time scales, implying that an additional noise is required for regularization.

We consider a first order convex optimization-based framework for computing M-estimators under local differential privacy (LDP) constraints. We assume a group of users communicates with an untrusted central server that learns the parameters of an underlying model. Furthermore, each user contributes multiple data points to the server. Contrary to most works that aim to protect a single data point, we focus on user-level privacy, that aims to protect the entire set of data points belonging to a user. Our main algorithm is a noisy version of the standard gradient descent algorithm, combined with a user-level LDP mean estimation procedure to privately compute the average gradient across users at each step.  We characterize the rate of convergence for parameter estimation under user-level local privacy constraints, and show that it improves with the number of users and the number of samples per user.

An essential feature in modern data science, especially in machine learning as well as high-dimensional statistics, are large sample sizes and large parameter space dimensions. As a consequence, the design of methods for uncertainty quantification is characterized by a tension between numerically feasible and efficient algorithms and approaches which satisfy theoretically justified statistical properties.

In this talk we discuss a Bayesian MCMC-based method with a stochastic Metropolis-Hastings as a potential solution. By calculating acceptance probabilities on batches, a stochastic Metropolis-Hastings step saves computational costs, but reduces the effective sample size. We show that this obstacle can be avoided by a simple correction term. We study statistical properties of the resulting stationary distribution of the chain if the corrected stochastic Metropolis-Hastings approach is applied to sample from a Gibbs posterior distribution in a nonparametric regression setting. Focusing on deep neural network regression, we prove a PAC-Bayes oracle inequality which yields optimal contraction rates and we analyze the diameter and show high coverage probability of the resulting credible sets. 

The talk is based on joint work with Sebastian Bieringer, Gregor Kasieczka and Maximilian F. Steffen.

The subject of our analysis are discrete time processes that take their values in a (separable)

Hilbert space, so called functional time series. A variety of statistics for functional time series allows for a

representation as weighted average of corresponding periodogram operators over the frequency domain.

We study consistency and asymptotic normality of such spectral mean estimators under mild

assumptions. As a by-product, we derive some basic first and second order properties of periodogram

operators as well. We show that weak convergence of spectral mean estimators can be deduced from the

(joint) weak convergence of the sample autocovariance operators. The latter is established for a large

class of weakly dependent functional time series, which admit expansions as Bernoulli shifts and the

weak dependence is quantified by the condition of L^4-m-approximability. In order to facilitate the

applicability of our results, we also propose a hybrid-bootstrap scheme, which combines a bootstrap-

approach with a subsampling method to approximate the gaussian limit of spectral mean estimators.

For functional data, it is often recommended to use dimension reduction or regularization techniques. But it is also possible to use the full functional information. We will discuss how the classical CUSUM test statistics and the Wilcoxon type statistic for the detection of change-points in time series can be generalized to observations that take values in a Hilbert-space. In order to obtain critical values, we propose to use Bootstrap methods.

Achieving a quantitative understanding and modelling of the dynamics of pedestrian crowds is not only a primary societal need, but also an outstanding fundamental challenge connected with the statistical physics of active flowing matter. Developments in automated vision over the last 15 years have given us increasing possibilities to measure the crowd behaviour at larger and larger spatial scales and with finer and finer details. We can now have access to individual tracking and biomechanics features in real-life conditions, anonymously, and even at high densities. Data acquisition with 24/7/365 schedules enables datasets including millions of trajectories, resolving common and rare events. These open big-data investigations aiming at statistical features emerging from universal physical properties of pedestrian dynamics.    

In this contribution, I will discuss some of the current possibilities in real-life data acquisition at large scales. Special attention will be on our works on 3D depth map data. I will show how large datasets enable statistical investigations from hydrodynamic aspects (probabilistic fundamental diagrams) to conditioned wayfinding statistics, from dynamical fluctuations to geometry aspects.

I'll present our current modelling works, hinging on bottom-up Langevin-like differential approaches, top-down generative AI, and in between the two. On all these points I will share my view on current and forthcoming opportunities and challenges.

We study anytime-valid (AV) tests for the presence of anomalies in a large

  number of data streams. AV tests monitor test martingales continously and

  their performance is judged using log-optimality criteria. We propose a

  framework to assess AV tests for our problem and show the fundamental limits

  of detection in a Gaussian-location instance of the problem. These results are

  related to but not implied by existing results for fixed-sample tests.

  Additionally, in the Gaussian-location setting, we show tests that are optimal

  even when the parameters of the problem are unknown. The methods that we

  develop set the benchmarks for more general optimal AV methods for sparse

  anomaly detection.

Generative Adversarial Networks (GANs) have attracted much attention since their introduction by Goodfellow el al. (2014), initially due to impressive results in the creation of photorealistic images. Meanwhile, the areas of application have expanded far beyond this, and GANs serve as a prototypical example of the rapidly developing experimental and theoretical research area of generative models. The statistical literature focuses mainly on Wasserstein GANs and their generalizations, which allow for good dimension reduction properties. Statistical results for Vanilla GANs, the original optimization problem, are still rather limited and require assumptions such as smooth activation functions and equal dimensions of the latent space and the ambient space. To bridge this gap, we draw a connection from Vanilla GANs to the Wasserstein distance. In doing so, existing results for Wasserstein GANs can be extended to Vanilla GANs. In particular, we obtain an oracle inequality for Vanilla GANs in Wasserstein distance. The assumptions of this oracle inequality are designed to be satisfied by commonly used network architectures, such as feedforward ReLU networks. By providing a quantitative result for the approximation of a Lipschitz function by a feedforward ReLU network with bounded Hölder norm, we conclude a convergence rate for Vanilla GANs.