Machine Learning Seminar Series

Abstract: Given a nonnegative matrix X and a factorization rank r, nonnegative matrix factorization (NMF) approximates the matrix X as the product of a nonnegative matrix W with r columns and a nonnegative matrix H with r rows. NMF has become a standard linear dimensionality reduction technique in data mining and machine learning. In this talk, we will first briefly introduce NMF, and why it is useful in various applications. Then, we address the issue of non-uniqueness of NMF decompositions, also known as the identifiability issue, which is crucial in many applications. We discuss three key NMF models that allow us to obtain unique NMFs, namely, separable NMF, minimum-volume NMF, and sparse NMF. We also discuss how the factors (W,H) in such models can be computed. We illustrate these results on facial feature extraction, blind hyperspectral unmixing, and topic modeling.

In recent years, the fundamental limits of distributed/federated learning have been studied under many statistical models, but often in the setting of horizontally partitioning, where data sets share the same feature space but differ in samples. Nevertheless, vertical federated learning, where data sets differ in features, have been in use in finance and medical care. In this talk, we consider a natural distributed nonparametric estimation problem with vertically partitioned datasets. Under a given budget of communication cost or information leakage constraint, we determine the minimax rates for estimating the density at a given point, which reveals that interactive protocols strictly improves over one-way protocols. Our novel estimation scheme in the interactive setting is constructed by carefully identifying a set of auxiliary random variables. The result also implies that interactive protocols strictly improve over one-way for biased binary sequences in the Gap-Hamming problem. (arXiv 2107.00211)

In this talk, we will focus on the emerging field of (adversarially) robust machine learning. The talk will be self-contained and no particular background on robust learning will be needed. Recent progress in this field has been accelerated by the observation that despite unprecedented performance on clean data, modern learning models remain fragile to seemingly innocuous changes such as small, norm-bounded additive perturbations. Moreover, recent work in this field has looked beyond norm-bounded perturbations and has revealed that various other types of distributional shifts in the data can significantly degrade performance. However, in general our understanding of such shifts is in its infancy and several key questions remain unaddressed.

Qizhi He is an Assistant Professor in the Department of Civil, Environmental, and Geo- Engineering at the University of Minnesota. He received his M.A. in applied mathematics and Ph.D. in structural engineering and computational science from UC San Diego, in 2016 and 2018, respectively. Afterwards, he was a Postdoctoral Research Associate at Pacific Northwest National Laboratory (PNNL), where he developed scientific machine learning methods for modeling flow and transport processes in porous media. His current research interests lie at the intersection of computational mechanics, materials modeling, and data-driven computing, with a focus on advancing data-driven machine learning enabled computational tools to predict mechanics of complex multiphysical processes and improve our fundamental understanding of multiscale materials and structures in engineered and natural systems.

Datasets are mathematical objects (e.g., point clouds, matrices, graphs, images, fields/functions) that have shape. This shape encodes important knowledge about the system under study. Topology is an area of mathematics that provides diverse tools to characterize the shape of data objects. In this work, we study a specific tool known as the Euler characteristic (EC). The EC is a general, low-dimensional, and interpretable descriptor of topological spaces defined by data objects. We revise the mathematical foundations of the EC and highlight its connections with statistics, linear algebra, field theory, and graph theory. We discuss advantages offered by the use of the EC in the characterization of complex datasets; to do so, we illustrate its use in different applications of interest in chemical engineering such as process monitoring, flow cytometry, and microscopy. We show that the EC provides a descriptor that effectively reduces complex datasets and that this reduction facilitates tasks such as visualization, regression, classification, and clustering.

I will be talking about our recent work on better understanding attention. Attention-based architectures have become ubiquitous in machine learning, yet our understanding of the reasons for their effectiveness remains limited. We show that self-attention possesses a strong inductive bias towards "token uniformity". Specifically, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to a rank-1 matrix. On the other hand, skip connections and MLPs stop the output from degeneration. Along the way, we develop a useful decomposition of attention architectures. This is joint work with Jean-Baptiste Cordonnier and Andreas Loukas.

Our paper and code are available online:

https://arxiv.org/abs/2103.03404;

https://github.com/twistedcubic/attention-rank-collapse.