Tel Aviv University Department of Statistics and Operations Research
Seminar takes place on Tuesdays 10:30-11:30 at Schriber 309 (TAU map ) unless noted otherwise .
For questions or to subscribe to emailing list: danielnevo at tauex.tau.ac.il
Scroll to bottom for a shareable calender
Second Semester (press for details)
Title: Showcasing the Statistical Lab
Abstract:
The Statistical Lab comprises a team of statistical experts who provide consulting services to researchers. Our lab collaborates with faculty members, graduate students, hospitals, and the public sector. Over the past 3 years, we have successfully completed more than 100 projects, and our ongoing objective is to further expand our workload. In this presentation, we will showcase two research studies in which the lab actively participated, along with an overview of the challenges and methodologies we employ in other projects as well.
Title: Predictive Analytics and Sensitivity Analysis of Association Rules
Abstract:
Modern organizations generate a large amount of unstructured textual transaction data on a daily basis. These include logbooks of systems and processes, technician reports of product failures and social media blogs. Transactions typically include semantic descriptors that require specialized methods of analysis. Exploring association rule (AR) is a powerful semantic data analytic technique used for extracting information from transaction databases. AR analysis was originally developed in shopping basket analysis where the combination of items in a shopping cart are evaluated. To generate an AR, the collection of more frequent itemsets, a set of two or more items—is first detected. Then, as a second step, all possible ARs are generated from each itemset in this group. The ARs are then ranked using measures of association such as support, confidence, and lift. These measures are labelled “measures of interest”. The R package “arules” provides more than a dozen such measures including the relative linkage disequilibrium (RLD) which normalizes classical Euclidean distances of the itemset from a surface of independence. The talk will show how to conduct a latent clustering and sensitivity analysis of ARs by clustering documents and repeatedly splitting data into a training and validation sets. These methods provide expanded interpretability and generalizability of AR analysis, even beyond the currently popular Rapid Keyword Extraction (RAKE) algorithm in Natural Language Processing.
References
Hahsler, M., Grun, B., and Hornik, K. (2005). arules – A computational environment for mining association rules and frequent item sets. Journal of Statistical Software, 14(15):1–25. ISSN 1548-7660. URL http://www.jstatsoft.org/v14/i15/.
Kenett, R. (1983). On an Exploratory Analysis of Contingency Tables. Journal of the Royal Statistical Society, series D, 32, pp. 395-403.
Kenett, R. S., and Salini, S. (2010). Measures of association applied to operational risks. Operational risk management: A practical approach to intelligent data analysis, 149-167, John Wiley and Sons.
Kenett, RS, Gotwalt, C, Freeman, L, Deng, X (2022). Self‐supervised cross validation using data generation structure. Applied Stochastic Models in Business and Industry, 38(5), 750-765.
Kenett RS, Zacks S, Gedeck P (2022). Modern Statistics: A Computer-Based Approach with Python, 1st edn. Springer-Birkhauser
Kenett, RS, and Gotwalt, C (2024). The analysis of association rules: latent class analysis, Statistical Analysis and Data Mining, in press.
Vives‐Mestres, M., Kenett, R. S., Thió‐Henestrosa, S., Martín‐Fernández, J. A. (2022). Measurement, selection, and visualization of association rules: A compositional data perspective: A Compositional Data perspective on Association Rules. Quality and Reliability Engineering International, 38(3), 1327-1339.
Title: Shavuot
Abstract:
Shavuot, also known as the Feast of Weeks, is a significant Jewish holiday celebrated in Israel and by Jews worldwide. It marks the end of a seven-week counting period following Passover. Shavuot commemorates the day the Torah was given to the Israelites at Mount Sinai, which is a pivotal event in Jewish history. Traditionally, it is observed by engaging in all-night study sessions of the Torah and other religious texts, reflecting the devotion and eagerness of the Israelites to receive the Torah. In Israel, the holiday also embraces agricultural aspects, with people decorating their homes and synagogues with greenery and flowers to symbolize the spring harvest. Dairy foods like cheesecake and blintzes are popularly consumed during Shavuot, symbolizing the land of milk and honey and the purity associated with the Torah. Shavuot is a public holiday in Israel, characterized by religious observance and cultural celebrations.
Title: Connections between Benign Overfitting and Local Theory of Banach Spaces
Abstract:
We investigate the statistical performance of ``minimum norm'' interpolators in non-linear regression under additive Gaussian noise. Specifically, we focus on norms that satisfy either 2-uniform convexity or the cotype 2 property -- these include inner-product spaces, \ell_{p} norms, and W_{p} Sobolev spaces for $1 \leq p \leq 2$. Our approach leverages tools from the local theory of finite dimensional Banach spaces, and, to the best of our knowledge, it is the first to study non-linear models that are ``far'' from Hilbert spaces. As an application of our approach, we prove optimal bounds for $\ell_p$ over-parametrized linear regression under sub-Gaussian covariates.
Title: TBA
Abstract:
TBA
Title: The real price of bandit information in multiclass classification
Abstract:
In PAC multiclass classification, examples are labeled with one of K possible labels. The learner receives a labeled training set and must return a hypothesis from a given hypothesis class H. The goal is to return a hypothesis whose error is at most \epsilon greater than that of the best hypothesis in H with probability 1-\delta. The minimum number of required samples defines the sample complexity of the problem.
In the bandit version, examples arrive one at a time. The learner guesses their label and only observes whether the guess was correct. This limited feedback increases the sample complexity, and this increment is known as the "price of bandit information." Classic works bound this increment by an additional factor of K, but whether this bound is tight remains an open question, with no nontrivial lower bound existing for this setting.
In our work, we investigate the true price of bandit information.
Online Setting: Our first study focuses on the online setting, where the aim is to bound regret—the difference between the number of mistakes made by our algorithm and the best hypothesis in hindsight. Prior work shows that regret can be bounded by \sqrt{K T \log |H|}, where T is the time horizon, compared to \sqrt{T \log |H|} when labels are fully observed. We improve this bound to \min\{|H| + \sqrt{T}, \sqrt{K T \log |H|}\}, providing matching upper and lower bounds thus establishing tightness up to logarithmic factors. Our lower bounds indicate that regret scales with the number of labels K, confirming an unavoidable price of bandit information as an additional factor of K.
PAC Setting: In our second study, we address the PAC setting and propose an algorithm with a sample complexity of (poly(K) + 1/\epsilon^2)\log(|H|/\delta). We demonstrate the implementation of our algorithm in polynomial time, given efficient empirical risk minimization over the hypothesis class. Surprisingly, we find that in the PAC setting, for sufficiently small \epsilon, there is no price for bandit information. This result reveals a significant gap between the price of bandit feedback in terms of regret and sample complexity, challenging conventional expectations.
Title: Semi-Supervised Learning with application to Generalized-Linear-Regression, Deep-Learning, and Interpolators
Abstract:
In numerous applications in the domain of machine learning, acquiring labeled data to train a supervised model is often more challenging than obtaining unlabeled data. Thus, there is a compelling reason to leverage the unlabeled data to enhance the predictive accuracy of a supervised learning model through the implementation of a suitable semi-supervised learning (SSL) strategy. Traditional studies indicate that SSL may not enhance linear models, especially when they are well specified. However, modern research shows that SSL methods can empirically improve state-of-the-art supervised models in real-world tasks.
In this work, we present an SSL methodology that improves the prediction performance of supervised regression models. The main idea is to design a mechanism for integrating the unlabeled data, including a \textit{mixing} parameter $\alpha$, controlling the weight given to the unlabeled data. Focusing on Generalized Linear Models (GLM), we analyze the characteristics of our SSL estimators, and prove that in all cases, it is invariably beneficial to integrate the unlabeled data with some nonzero mixing ratio $\alpha>0$, in terms of predictive performance.
The effectiveness of our methodology in delivering substantial improvement compared to the standard supervised models, is demonstrated empirically through extensive simulation, in a manner that supports the theoretical analysis. We also demonstrate the applicability of our methodology to improve more complex models, such as deep neural networks, in real-world regression tasks.
Title: Data Science vs. Statistics?
Abstract:
The profession of Statistics has a long and honorable tradition of introspection and contemplation. Dating back to e.g. Tukey's "The Future of Data Analysis" from 1962, statisticians have openly and publicly deliberated where the field is, and where it is going. Such introspection seems to be urgent now. For over a decade, it seems that the success of statistics in the data science marketplace has been inversely proportional to the amount of data and compute available. This is a good time to ask: Who is the Digital-Age Statistician? What is her role in data science, today and tomorrow? Building on Leo Breiman's stark, prophetic warning to statistics in "The Two Cultures" (2001), which claimed that the future (our present) is best understood along a fault line between two radically different worldviews, I will present a perspective on the fast-changing relative positions of the fields of machine learning, statistics, data science and applied mathematics in the digital age.
Joint work with David Donoho
Title: Object detection under the linear subspace model
Abstract:
Detecting unknown objects in noisy data is a key problem in many fields of science. A common model for the unknown objects is the linear subspace model, which assumes that the objects can be expanded in some known basis (such as the Fourier basis). In this talk, I will present an object detection algorithm that under the linear subspace model is asymptotically guaranteed to find all objects while making only a small percentage of false discoveries. We demonstrate our derivations for the problem of particle picking in cryo-electron microscopy.
Title: An introduction to Smoluchowski processes and some related statistical problems.
Abstract:
Suppose that particles are randomly distributed in a multi-dimensional Euclidean space, and they are subject to
identical stochastic motion independently of each other. The Smoluchowski process describes fluctuations of the number
of particles in an observation region over time. This class of processes, was originally motivated by the
Einstein–Smoluchowski theory in statistical physics. In addition, it has some interesting relations to queueing theory,
branching processes, etc. In this talk, I will review a recent work https://arxiv.org/pdf/2108.06954 (joint
with Alexander Goldenshluger) in which we studied the properties of the Smoluchowski processes with some
related statistical problems.
Title: TBA
Abstract:
TBA
First Semester (press for details)
Title: Spectral Properties of Elementwise-Transformed Spiked Matrices
Abstract:
From principal component analysis (PCA) to covariance estimation to factor analysis, spiked matrices of the form Y = X + Z are widely used to model high-dimensional data with latent low-rank structure. Here, X is a low-rank signal matrix and Z is a noise matrix. In this talk, we extend spiked matrix results to the model Y = f(X+Z), where f is a function applied elementwise. This model includes certain types of missing data, truncated data, unsigned data, and binomial data with latent low-rank structure.
We find that principal component analysis is powerful for recovering signal under highly nonlinear or discontinuous transformations. Specifically, we find a phase transition in high dimensions: for signal-to-noise ratios above a sharp threshold---depending on f, the distribution of elements of Z, and the aspect ratio of the data---the principal components of Y (partially) recover those of X. Below this threshold, the principal components of Y are asymptotically orthogonal to the signal. In contrast, in the standard setting where X + Z is observed directly, the analogous phase transition (the Baik-Ben Arous-Péché threshold) depends only on the aspect ratio of the data. Similar phenomena occur with Y, X square and symmetric and Z a (generalized) Wigner matrix.
Title: Causally-driven ML for Out-of-distribution Generalization
Abstract:
The reliance of machine learning (ML) classifiers on spurious correlations can lead to poor generalization at deployment, raising concerns about their use in safety-critical domains such as healthcare. In this talk, I will show how by understanding causal relationships in data we can enforce independence from spurious features, thus improving out-of-distribution (OOD) performance. First, I will discuss why prominent methods for enforcing independence often struggle on real-world data - their complex learning objectives and inefficient use of training samples. Linking OOD performance with model calibration, I will then argue that calibration across multiple datasets results in classifiers that are easier to learn and are provably free of spurious correlations. Then, to improve sample complexity, I will propose methods for counterfactual data augmentation to simulate interventions on these features. Finally, I will demonstrate, through experiments on learning predictors of clinical diagnoses from medical narratives, that our methods for simulating interventions improve OOD accuracy.
Title: Fisher's PET
Abstract:
Shepp and Vardi created in 1982 a friendly Poisson version of the EM algorithm to estimate the charge function of a tumor via PET scan of tissue, still in use today. Dempster, Laird and Rubin introduced in 1977 the EM algorithm for MLE point estimates of parameters, without Fisher information and the ensuing standard errors, P-values and confidence intervals. These were presented in the 1980s in one form or another, by me in particular. During the past few months the numerical approximations in these methods converged for PET scan to a closed form formula. The talk will cover a refresher on EM, the idea of incorporating an estimate of the Fisher information matrix purely in EM terms, with emphasis on PET and the closed form formula. If the tissue is 200 by 200 pixels, Fisher’s matrix is 40000 by 40000, allowing inversion of small minors only. This subject needs further study.
Users got used to relate to PET as actual pictures and not as a list of estimated parameters. I want to foster a discussion on making statistical error calculations practical, and how to convince the industry that it is worthwhile to give it a try.
Title: Risk-Controlling Model Selection Combining Multi-Objective Optimization and Statistical Testing
Abstract:
With the great promise of the data-science revolution, a major open question is whether the underlying models are reliable and trustworthy, especially when deployed in high-stakes applications. To overcome these challenges statistical tools can be harnessed for uncertainty quantification and control of complex machine learning models. Specifically, the configuration of models is often controlled by a combination of tunable hyperparameters, which give rise to different performance behaviors with respect to multiple, often conflicting objectives, such as accuracy, fairness, robustness, and inference cost. I will present a new framework for selecting model configurations that provably satisfy multiple explicit and simultaneous statistical guarantees, while also optimizing any number of additional unconstrained objectives. The method combines multi-objective optimization and statistical testing, creating a framework that is both computationally and statistically efficient, even when the configuration space is combinatorially large. To further align the two stages of optimization and testing, I will propose a new Bayesian optimization method that is steered towards an efficient testing strategy, enabling reliable tuning of expensive hyperparameter combinations that require model retraining. We demonstrate the effectiveness of our approach on a range of tasks with multiple desiderata, including low error rates, equitable predictions, handling spurious correlations, managing rate and distortion in generative models, and reducing computational costs.
Title: Estimation of the Number Needed to Treat, the Number Needed to be Exposed, and the Exposure Impact Number with Instrumental Variables
Abstract:
The Number needed to treat (NNT) is an efficacy index defined as the average number of patients needed to treat to attain one additional treatment benefit. In observational studies, specifically in epidemiology, the adequacy of the populationwise NNT is questionable since the exposed group characteristics may substantially differ from the unexposed. To address this issue, groupwise efficacy indices were defined: the Exposure Impact Number (EIN) for the exposed group and the Number Needed to be Exposed (NNE) for the unexposed. Each defined index answers a unique research question since it targets a unique sub-population. In observational studies, the group allocation is typically affected by confounders that might be unmeasured. The available estimation methods that rely either on randomization or the sufficiency of the measured covariates for confounding control will result in inconsistent estimators of the true NNT (EIN, NNE) in such settings. Using Rubin's potential outcomes framework, we explicitly define the NNT and its derived indices as causal contrasts. Next, we introduce a novel method that uses instrumental variables to estimate the three aforementioned indices in observational studies. We present two analytical examples and a corresponding simulation study. The simulation study illustrates that the novel estimators are consistent, unlike the previously available methods, and their confidence intervals meet the nominal coverage rates. Finally, a real-world data example of an analysis of the effect of vitamin D deficiency on the mortality rate is presented.
Title: TBA
Abstract:
TBA
Title: Exact confidence intervals for the mixing distribution from binomial mixture distribution samples
Abstract:
We present methodology for constructing pointwise confidence intervals for the cumulative distribution function
and the quantiles of mixing distributions on the unit interval from binomial mixture distribution samples. No assumptions
are made on the shape of the mixing distribution. The confidence intervals are constructed by inverting exact tests of
composite null hypotheses regarding the mixing distribution. Our method may be applied to any deconvolution approach
that produces test statistics whose distribution is stochastically monotone for stochastic increase of the mixing distribution.
We propose a hierarchical Bayes approach, which uses finite Polya Trees for modelling the mixing distribution, that provides
stable and accurate deconvolution estimates without the need for additional tuning parameters. Our main technical result
establishes the stochastic monotonicity property of the test statistics produced by the hierarchical Bayes approach.
Leveraging the need for the stochastic monotonicity property, we explicitly derive the smallest asymptotic confidence intervals
that may be constructed using our methodology. Raising the question whether it is possible to construct smaller confidence
intervals for the mixing distribution without making parametric assumptions on its shape. This is joint work with Pallavi Basu and Barak
Brill.
Title: Negative Control Tests for Instrumental Variable Designs
Abstract:
Studies using instrumental variables (IV) often assess the validity of their identification assumptions using falsification tests. However, these tests are often carried out in an ad-hoc manner, without theoretical foundations. In this paper, we establish a theoretical framework for negative control tests, the predominant category of falsification tests for IV designs. These tests are conditional independence tests between negative control variables and either the IV or the outcome. We introduce a formal definition for threats to IV exogeneity as alternative path variables and characterize the necessary conditions that proxy variables for such unobserved threats must meet to serve as negative controls. The theory highlights prevalent errors in the implementation of negative control tests and how they could be corrected. The theory can also be used to design new falsification tests by identifying appropriate negative control variables, including currently underutilized types, and suggesting alternative statistical tests. The theory shows that all negative control tests assess IV exogeneity. However, some commonly used tests simultaneously evaluate the 2SLS functional form assumptions. Lastly, we show that while negative controls are useful for detecting biases in IV designs, their capacity to correct or quantify such biases requires additional non-trivial assumptions.
Joint work with Oren Danieli, Dan Zeltzer (TAU Econ), Bar Weinstein and Itai Walk (TAU Stat)
Title: What Makes Data Suitable for Deep Learning?
Abstract:
Deep learning is delivering unprecedented performance when applied to various data modalities, yet there are data distributions over which it utterly fails. The question of what makes a data distribution suitable for deep learning is a fundamental open problem in the field. In this talk I will present a recent theory aiming to address the problem via tools from quantum physics. The theory establishes that certain neural networks are capable of accurate prediction over a data distribution if and only if the data distribution admits low quantum entanglement under certain partitions of features. This brings forth practical methods for adaptation of data to neural networks, and vice versa. Experiments with widespread models over various datasets will demonstrate the findings. An underlying theme of the talk will be the potential of physics to advance our understanding of the relation between deep learning and real-world data.
The talk is based on NeurIPS 2023 papers co-authored with my students Noam Razin, Yotam Alexander, Nimrod De La Vega and Tom Verbin.
Title: TBA
Abstract:
TBA
Title: Completing large matrices with only few observed entries: A provable one-line algorithm
Abstract:
Suppose you observe very few entries from a large matrix. In many applications, you can safely assume the matrix is (approximately) low rank. How can you use this knowledge to complete the missing entries? In this talk, I will present a novel yet surprisingly simple method to solve the problem. Our method is shown to recover the matrix in the challenging settings of very few observations and/or ill-conditioning, where many popular methods fail. We also prove strong theoretical guarantees for our method, some of which improve upon the state of the art. Finally, I will discuss how one can complete a matrix by incorporating extra knowledge beyond its observed entries, a problem known as inductive matrix completion. The talk is based on two joint papers with Boaz Nadler, published in SIMODS and ICML.
Second Semester (press for details):
Add calender for non-google calenders: link to ics file
Past years:
Talks from previous years can be found in the tabs on the top right of this page.
Talks from before the 2021/2022 academic year can be found at this link.
