Title: Weighted least squares: better generalization through smoother interpolation
Abstract: The double descent phenomenon refers to the observation that modern neural network optimization problems often exhibit performance that first improves, then gets worse, and then improves again with increasing model size, seemingly contradicting the classical wisdom that increasing model size leads to overfitting.
We show that the double descent curve is connected to the implicit bias of gradient descent toward smoother and smoother interpolants. We derive precise expressions for the generalization error using a random Fourier series model for smooth functions. We then argue through this model and numerical experiments that normalization methods in deep learning such as weight normalization improve generalization in overparameterized neural networks by implicitly encouraging smooth interpolants. This is work with Yuege (Gail) Xie, Holger Rauhut, and Hung-Hsu Chou.
Title: Variation modeling meets learning
Abstract: In this talk, I will show how to use learning techniques to significantly improve variation models (also known as energy minimization models). I start by showing that for the simplest models such as total variation, one can greatly improve the accuracy of the numerical approximation by learning the "best" discretization within a class of consistent discretizations. I will then show how such models can be further extended and improved to provide state-of-the-art results for a number of important image reconstruction problems. Finally, I will show that learning can also be used in combination with discrete energy minimization problems, leading to state-of-the-art results in stereo, optical flow, and segmentation.
Title: Image Restoration and Beyond
Abstract: We are living in the era of big data. The discovery, interpretation and usage of the information, knowledge and resources hidden in all sorts of data to benefit human beings and to improve everyone's day to day life is a challenge to all of us. The huge amount of data we collect nowadays is so complicated, and yet what we expect from it is so much. This provides many challenges and opportunities to many fields. As images are one of the most useful and commonly used types of data, in this talk, we start from reviewing the development of the wavelet frame (or more general redundant system) based approach for image restoration. We will observe that a good system for any data, including images, should be capable of effectively capturing both global patterns and local features. One of the examples of such system is the wavelet frame. We will then show how models and algorithms of wavelet frame based image restoration are developed via the generic knowledge of images. Then, the specific information of a given image can be used to further improve the models and algorithms. Through this process, we shall reveal some insights and understandings of the wavelet frame based approach for image restoration and its connections to other approaches, e.g. the partial differential equation based methods. Finally, we will also show, by many examples, that ideas given here can go beyond image restoration and can be used to many other applications in data science.
Title: Classifying stroke from electric data by nonlinear Fourier analysis and machine learning
Abstract: There are two main types of stroke: ischemic (blood clot preventing blood flow to a part of the brain) and hemorrhagic (bleeding in the brain). A portable “stroke classifier" would be a life-saving equipment to have in ambulances. Can one use Electrical Impedance Tomography (EIT) for stroke classification? EIT aims to recover the electric conductivity inside a domain from electric boundary measurements, and the two strokes differ in conductivity. A new property of Complex Geometric Optics (CGO) solutions for EIT is presented, showing that a one-dimensional Fourier transform in the spectral variable yields profile functions similar to those of parallel-beam X-ray tomography. In practical imaging, measurement noise causes strong blurring in the profile functions. However, machine learning (ML) algorithms can be used to overcome this problem. It turns out that simulated strokes are classified by ML much more accurately from nonlinear Fourier features (=CGO profiles) than from the unprocessed EIT measurements.
Title: An Inner-Outer Iterative Method for Edge Preservation in Image Restoration and Reconstruction
Abstract: We present a new inner-outer iterative algorithm for edge enhancement in imaging restoration and reconstruction problems. At each outer iteration, we formulate a Tikhonov-regularized problem where the penalization is expressed in the 2-norm and involves a regularization operator designed to improve edge resolution as the outer iterations progress, through an adaptive process. An efficient hybrid regularization method is used to project the Tikhonov-regularized problem onto approximation subspaces of increasing dimensions (inner iterations), while conveniently choosing the regularization parameter via well-known techniques, such as the discrepancy principle or the L-curve criterion, applied to the projected problem. This procedure results in an automated algorithm for edge recovery that does not involve regularization parameter tuning by the user, nor repeated calls to sophisticated optimization algorithms. We demonstrate the value of our approach on applications in X-ray CT image reconstruction and in image deblurring and show that it can be computationally much more attractive than other well-known strategies for edge preservation, while providing solutions of greater or equal quality.
This is joint work with Silvia Gazzola, James G. Nagy, Oguz Semerci, Eric L. Miller.
Title: Polar deconvolution of mixed signals
Abstract: The signal demixing problem seeks to separate the superposition of multiple signals into its constituent components. We model the superposition process as the polar convolution of atomic sets, which allows us to use the duality of convex cones to develop an efficient two-stage algorithm with sublinear iteration complexity. If the signal measurements are random, the polar deconvolution approach stably recovers low-complexity and mutually-incoherent signals with high probability and with optimal sample complexity. Numerical experiments on both real and synthetic data confirm the theory and efficiency of the proposed approach.
Joint work with Zhenan Fan, Halyun Jeong, and Babhru Joshi at the University of British Columbia.
Title: Artifact suppression in X-ray CT images
Abstract: X-ray Computed Tomography (CT) is one of the most powerful techniques for visualizing internal structure of a scanned object. Recently, it has been widely used in the field of industry as well as in the field of medicine. However, there are various kinds of artifacts causing severe degradation of CT images. Therefore, developing an accurate and efficient artifact suppression method is becoming an important issue in X-ray CT imaging and its applications. In this talk, we will look at the mathematics of X-ray CT and artifacts in CT images. Moreover, we will deal with some recent works to suppress them such as metal artifacts and scattering artifacts.
Title: Curve Based Approximation of Measures on Manifolds
Abstract: The approximation of probability measures on compact metric spaces and in particular on Riemannian manifolds by atomic or empirical ones is a classical task in approximation and complexity theory with a wide range of applications. Instead of point measures we are concerned with the approximation by measures supported on Lipschitz curves. Special attention is paid to pushforward measures of Lebesgue measures on the unit interval by such curves. Using the discrepancy as distance between measures, we prove optimal approximation rates in terms of the curve's length and Lipschitz constant. Having established the theoretical convergence rates, we are interested in the numerical minimization of the discrepancy between a given probability measure and the set of push-forward measures of Lebesgue measures on the unit interval by Lipschitz curves. We present numerical examples for measures on the 2- and 3-dimensional torus, the 2-sphere, the rotation group on R3 and the Grassmannian of all 2-dimensional linear subspaces of R4. Our algorithm of choice is a conjugate gradient method on these manifolds which incorporates second-order information. For efficient gradient and Hessian evaluations within the algorithm, we approximate the given measures by truncated Fourier series and use fast Fourier transform techniques on these manifolds.
Title: Bayesian reimaging of sparsity in inverse problems
Abstract: The recovery of sparse generative models from few noisy measurements is a challenging inverse problem with applications to several areas, from image reconstruction to dictionary learning. In the Bayesian framework, sparsity can be interpreted as an a priori belief about the solutions, and suitable priors can be designed to promote sparsity in the computed solution. In this talk we present an efficient computational framework for Bayesian sparse inverse solvers that encodes the sparsity in terms of hierarchical prior models whose parameters can be set according to the level of sparsity expected and the sensitivity of the data to the solution. The computational procedure can be organized as an inner-outer iteration scheme, where a weighted linear least squares problem is solved in the inner iteration and the outer iteration update dynamically the scaling weights. When the least squares problems are solved approximately by the Conjugate Gradient method for least squares (CGLS) equipped with a suitable stopping rule, typically the number of CGLS iterations quickly converges to the cardinality of the support, thus providing an automatic model reduction. We will show computed examples illustrating the performance of the Bayesian approach to sparsity in imaging and dictionary learning applications.
Title: Stochastic primal dual splitting algorithms for convex and nonconvex composite optimization in imaging
Abstract: Primal dual splitting algorithms are largely adopted for composited optimization problems arising in imaging. In this talk, I will present stochastic extensions of some popular composite optimization algorithms in both convex and nonconvex settings and their applications to image restoration and statistical learning problems. The first class of algorithms is designed for convex linearly composite problems by combining stochastic gradient with the so-called primal dual fixed point method (PDFP). As a natural generalization of proximal stochastic gradient types methods, we proposed stochastic PDFP (SPDFP) and its variance reduced version SVRG-PDFP which do not require subproblem solving. The convergence and convergence orders are established for the proposed algorithms based on some standard assumptions. The numerical examples on graphic Lasso, graphics logistic regressions and image reconstruction are provided to demonstrate the effectiveness of the proposed algorithm. In particular, we observe that for large scale image reconstruction problems, SVRG-PDFP exhibits some advantages in terms of accuracy and computation speed, especially in the case of relatively limited high performance computing resource The second class of algorithms is based on Alternating direction method of multipliers (ADMM) for nonconvex composite problems. In particular, we study the ADMM method combined with a class of variance reduction gradient estimators and established the global convergence of the sequence and convergence rate under the assumption of Kurdyka-Lojasiewicz (KL) function.
Title: The Half-space Matching Method or how to solve scattering in complex media
Abstract: This is a joint work with Anne-Sophie Bonnet-Ben Dhia (POEMS), Patrick Joly (POEMS), Yohanes Tjandrawidjaja (TU Dortmund), Antoine Tonnoir (INSA Rouen). We are interested in the scattering of time-harmonic waves in infinite complex media. The complexity of the media comes from the nature of the equations (Maxwell's or elasticity equations), its physical characteristics (periodic or anisotropic coefficients) and/or even its geometry (infinite 2D or 3D media or 3D plates). Solving time harmonic scalar waves equations in infinite homogeneous media is an old topic (see for instance the review paper [1]) and there exist several methods. They are all based on the natural idea of reducing the pure numerical computations to a bounded domain containing the perturbations (achieved using for instance Finite Element methods), coupled with, for example, integral equation techniques, transparent boundary conditions involving Dirichlet-to-Neumann operators or the Perfectly Matched Layer techniques. However it seems that all these methods either do not extend to complex media or do extend but with a tremendous computational cost. By contrast, our method is based on a simple and quite general idea: the solution of halfspace problems can be expressed thanks to its trace on the edge of the half-space, via the Fourier transform in the transverse direction in the homogeneous case or via the Floquet-Bloch Transform in the periodic case. The idea in 2D is then to split the whole domain into five parts:
- a square that includes the defect (and all the inhomogeneities) in which we will use a Finite Elements representation of the solution,
- and $4$ half-spaces, parallel to the four edges of the square in which the medium is "not-perturbed", i.e. homogeneous or periodic.
We have then to couple the several integral representations of the solution in the half-spaces with a Finite Element (FE) computation of the solution in the square. By ensuring that all these representations match in the intersections, we end up with a system of equations coupling the traces of the solution on the edges of the half-spaces with the restriction of the solution in the square. In the case of a dissipative medium, the continuous formulation is proved to be coercive plus compact, and the convergence of the discretization is ensured [2,3]. In this presentation, we present the method and its analysis on a toy problem and we will explain how it extends to more complex situations.
[1] D. Givoli. , Numerical methods for problems in infinite domains., Vol. 33. Elsevier, 2013.
[2]A. S. Bonnet-Ben Dhia, S. Fliss and A. Tonnoir, The halfspace matching method: A new method to solve scattering problems in infinite media, Journal of Computational and Applied Mathematics, 338, 44-68, 2018.
[3]A. S. Bonnet-Ben Dhia, S. Fliss and Y. Tjandrawidjaja, Numerical analysis of the Half-Space Matching method with Robin traces on a convex polygonal scatterer, hal-01793511, 2018.
Title: Bilevel learning methods in imaging
Abstract: Optimization techniques have been widely used for image restoration tasks, as many imaging problems may be formulated as energy minimization ones with the recovered image as the target minimizer. Recently, novel optimization ideas also entered the scene in combination with machine learning approaches, to improve the reconstruction of images by optimally choosing different quantities/functions of interest in the models. In this talk I will provide a review of the latest developments concerning the latter, with special emphasis on bilevel optimization techniques and their use for learning local and nonlocal image restoration models in a supervised manner.
Title: Nonlinear spectral decompositions in imaging and inverse problems
Abstract: This talk will describe the development of a variational theory generalizing classical spectral decompositions in linear filters and singular value decomposition of linear inverse problems to a nonlinear regularization setting in Banach spaces. We will discuss some applications in imaging and data science and discuss the computation of nonlinear eigenfunctions by gradient flows and power iterations.
Title: The Softmax function, Potts model and variational neural networks
Abstract: In this talk, we present our recent research on using variational models as layers for deep neural networks (DNNs). We use image segmentation as an example. The technique can also be used for high dimensional data classification as well. Through this technique, we could integrate many well-know variational models for image segmentation into deep neural networks. The new networks will have the advantages of traditional DNNs. At the same time, the outputs from the new networks can also have many good properties of variational models for image segmentation. We will present some techniques to incorporate shape priors into the networks through the variational layers. We will show how to design networks with spatial regularization and volume preservation. We can also design networks with guarantee that the output shapes from the network for image segmentation must be convex shapes/star-shapes. It is numerically verified that these techniques can improve the performance when the true shapes satisfy these priors. The ideas of these new networks is based on some relationship between the softmax function, the Potts models and the structure of traditional DNNs. We will explain this in detail which leads naturally to the newly designed networks.
This talk is based on joint works with Jun Liu, S. Luo and several other collaborators.
Eric Bonnetier, Université Grenoble-Alpes, France.
Luca Calatroni, CNRS, Université Côte d'Azur, France.
Raymond Chan, CityU, Hong Kong.
Fadil Santosa, Johns Hopkins University, USA
Carola-Bibiane Schönlieb, University of Cambridge, UK.