Deep learning approaches have been extensively studied as a fast and high-performance alternative for inverse problems. However, most existing deep learning works require matched reference data for supervised training, which are often difficult in inverse problems. To address this, we first derive a novel cycleGAN framework from optimal transport formulation of unsupervised learning problems, by simultaneously minimizing the statistical distances in both measurement and image domain. One of the most important advantages of this formulation is that depending on the knowledge of the forward operator, distinct variations of cycleGAN architecture can be derived: for example, one with two pairs of generators and discriminators, and the other with only a single pair of generator and discriminator. Even for the two generator cases, we show that the structural knowledge of the forward operator can lead to a simpler generator architecture which significantly simplifies the neural network training. Furthermore, to overcome the limitation of standard cycleGAN framework with two deep generators, we present a novel tunable CycleGAN architecture using a single generator. In particular, a single generator is implemented using adaptive instance normalization (AdaIN) layers so that the baseline generator can be switched between the forward and inverse generators by simply changing the AdaIN code. Thanks to the shared baseline network, the additional memory requirement and weight increases can be minimized, and the training can be done more stably even with small training data. Furthermore, by interpolating the AdaIN codes between the two domains, we can investigate continuous variation of the network output along the optimal transport path. We also provide various real world applications such as CT, MRI, ultrasound, phase retrieval, etc.

In this talk, we present a global landscape analysis of GANs (generative adversarial nets). We prove that a class of Separable-GANs (SepGAN), including the popular JS-GAN and hinge-GAN, have exponentially many bad strict local minima; in addition, they are mode-collapse patterns. We prove that relativistic pairing GANs (RpGANs) haver no bad basins. RpGAN can be viewed as an unconstrained variant of W-GAN: RpGAN keeps the "pairing" idea of W-GAN, but adds an upper bounded shell function (e.g. logistic loss in JS-GAN). The empirical benefit of RpGANs has been demonstrated by practitioners in, e.g., ESRGAN and realnessGAN. We demonstrate that the better landscape of RpGANs makes a difference in practice, with 2 lines of code change. We predict that RpGANs has a bigger advantage over SepGAN for high resolution data (e.g. LSUN 256*256), imbalanced data and narrower nets (1/2 or 1/4 width), and our experiments on real data verify these predictions.

In this talk, we offer an entirely “white box’’ interpretation of deep (convolutional) networks. In particular, we show how modern deep architectures, linear (convolution) operators and nonlinear activations, and parameters of each layer can be derived from the principle of rate reduction (and invariance). All layers, operators, and parameters of the network are explicitly constructed via forward propagation, instead of learned via back propagation. All components of such a network have precise optimization, geometric, and statistical meaning. There are also several nice surprises from this principled approach that shed new light on fundamental relationships between forward (optimization) and backward (variation) propagation, between invariance and sparsity, and between deep networks and Fourier analysis.

In this talk we will discuss the problem of classification in the presence of noisy annotations. When trained on noisy labels, deep neural networks have been observed to first fit the training data with clean labels during an early learning phase, before eventually memorizing the examples with false labels. We show that early learning and memorization are fundamental phenomena in high-dimensional classification tasks, proving that they occur even in simple linear models. Motivated by these findings, we develop a new technique for noisy classification tasks, which exploits the progress of the early learning phase.

Exciting progresses have been made in demystifying the efficacy of vanilla gradient methods in solving challenging nonconvex problems in statistical estimation and machine learning. In this talk, we discuss how the trick of preconditioning further boosts the convergence speed with minimal computation overheads through two examples: low-rank matrix estimation in statistical learning and policy optimization in entropy-regularized reinforcement learning. For low-rank matrix estimation, we present a new algorithm, called scaled gradient descent, that achieves linear convergence at a rate independent of the condition number of the low-rank matrix at near-optimal sample complexities for a variety of tasks. For policy optimization, we develop the first fast non-asymptotic convergence guarantee for entropy-regularized natural policy gradient methods in the tabular setting for discounted Markov decision processes. By establishing its global linear convergence at a near dimension-free rate, we provide theoretical footings to the empirical success of entropy-regularized natural policy gradient methods. Based on arXiv 2005.08898 and 2007.06558.

Panelists: Yuejie Chi, Tom Goldstein, Ambar Pal, Rene Vidal

We consider the problem of learning a union of subspaces from data corrupted by outliers. Classical approaches based on convex optimization are designed for subspaces whose dimension is small relative to the ambient dimension. Here we propose a non-convex approach called Dual Principal Component Pursuit (DPCP), which is specifically designed for subspaces whose dimension is close to the ambient dimension. We provide theoretical guarantees under which every global solution to DPCP is a vector in the orthogonal complement to one of the subspaces. We also study various optimization algorithms for solving the DPCP problem and analyze their convergence to a global minimum. Experiments show that the proposed methods are able to handle more outliers and higher relative dimensions than state-of-the-art methods.

Deep neural networks are often considered to be complicated "black boxes," for which a full systematic analysis is not only out of reach but also impossible. In this talk, which is based on ongoing joint work with Daniel Adam Roberts and Sho Yaida, I will make the opposite claim. Namely, that deep neural networks at initialization are exactly solvable models. Our approach applies to networks at finite width n and large depth L, the regime in which they are used in practice. A key point will be the emergence of a notion of "criticality," which involves a fine-tuning of model parameters (weight and bias variances). At criticality, neural networks are particularly well-behaved but still exhibit a tension between large values for n and L, with large values of n tending to make neural networks more like Gaussian processes and large values of L amplifying higher cumulants. Our analysis has a range of consequences such both for understanding feature learning and for optimal settings of hyperparameters such as weight/bias variances and learning rates.

We study the multiple manifold problem, a binary classification task modeled on applications in machine vision, in which a deep fully-connected neural network is trained to separate two low-dimensional submanifolds of the unit sphere. We provide an analysis of the one-dimensional case, proving for a simple manifold configuration that when the network depth L is large relative to certain geometric and statistical properties of the data, the network width n grows as a sufficiently large polynomial in L, and the number of i.i.d. samples from the manifolds is polynomial in L, randomly-initialized gradient descent rapidly learns to classify the two manifolds perfectly with high probability. Our analysis demonstrates concrete benefits of depth and width in the context of a practically-motivated model problem: the depth acts as a fitting resource, with larger depths corresponding to smoother networks that can more readily separate the class manifolds, and the width acts as a statistical resource, enabling concentration of the randomly-initialized network and its gradients. The argument centers around the “neural tangent kernel” of Jacot et al. and its role in the nonasymptotic analysis of training overparameterized neural networks; to this literature, we contribute essentially optimal rates of concentration for the neural tangent kernel of deep fully-connected networks, requiring width n 􏰀 L poly(d0) to achieve uniform concentration of the initial kernel over a d0-dimensional submanifold of the unit sphere Sn0−1, and a nonasymptotic framework for establishing generalization of networks trained in the “NTK regime” with structured data. The proof makes heavy use of martingale concentration to optimally treat statistical dependencies across layers of the initial random network. This approach should be of use in establishing similar results for other network architectures.

Joint work with Sam Buchanan and Dar Gilboa

Modern deep generative models like GANs, VAEs and invertible flows are achieving amazing results on modeling high-dimensional distributions, especially for images. We will show how they can be used to solve inverse problems by generalizing compressed sensing beyond sparsity, extending the theory of Restricted Isometries to sets created by generative models. We will present the general framework, new results and open problems in this space.

This talk will dissect the role of dimensionality in neural networks using scientific and empirical methods. First, I'll discuss the role of low dimensionality in learning, and present evidence for low dimensionality in complex image sets. We obtain this evidence by applying dimensionality estimators to sets of natural images (e.g., ImageNet) and discovering that their apparent dimensionality is surprisingly low. Furthermore, we observe that more data is needed to learn image manifolds of higher dimensionality. Second, I'll discuss the role of high-dimensionality in learning. Using visualization and empirical studies, I'll discuss the sharp/flat hypothesis, and present evidence that the curse of dimensionality in high dimensional parameter spaces plays an important role in network generalization.

Talk 8: A Game Theoretic Analysis of Additive Adversarial Attacks and Defenses

Speaker: Ambar Pal, Johns Hopkins University

Time: 4:10 pm - 4:20 pm

Abstract: Research in adversarial learning follows a cat and mouse game between attackers and defenders where attacks are proposed, they are mitigated by new defenses, and subsequently new attacks are proposed that break earlier defenses, and so on. However, it has remained unclear as to whether there are conditions under which no better attacks or defenses can be proposed. In this talk, I will describe a game-theoretic framework for studying attacks and defenses which exist in equilibrium. Under a locally linear decision boundary model for the underlying binary classifier, we will see that the Fast Gradient Method attack and a Randomized Smoothing defense form a Nash Equilibrium. We will see that this equilibrium defense can be approximated given finitely many samples from a data-generating distribution, and look at a generalization bound for the performance of this approximation.