Gal Vardi

I am a Senior Scientist (=Assistant Professor) in the Department of Computer Science and Applied Mathematics at the Weizmann Institute of Science. Prior to that, I was a postdoctoral researcher at TTI-Chicago (hosted by Nati Srebro), the Hebrew University (hosted by Amit Daniely), and Weizmann (hosted by Ohad Shamir). I completed my PhD at the Hebrew University, advised by Orna Kupferman.

My research focuses on theoretical machine learning, with an emphasis on deep-learning theory.

Boaz Nadler, Ohad Shamir, and I are co-organizing the Machine Learning and Statistics Seminar.

Email: gal.vardi at weizmann dot ac dot il

Papers

(* = Equal contribution or alphabetical order)

Provable Privacy Attacks on Trained Shallow Neural Networks

Guy Smorodinsky, Gal Vardi and Itay Safran

Preprint, 2024

Reconstructing Training Data From Real World Models Trained with Transfer Learning

Yakir Oz, Gilad Yehudai, Gal Vardi, Itai Antebi, Michal Irani and Niv Haim

Preprint, 2024

Transformers are almost optimal metalearners for linear classification

Roey Magen and Gal Vardi

To appear in NeurIPS 2025

Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processes

Itamar Harel, Yonathan Wolanowsky, Gal Vardi, Nathan Srebro and Daniel Soudry

To appear in NeurIPS 2025 (Selected as Spotlight)

Benign Overfitting in Single-Head Attention

Roey Magen*, Shuning Shang*, Zhiwei Xu, Spencer Frei, Wei Hu and Gal Vardi

To appear in NeurIPS 2025

A preliminary version appeared in M3L Workshop @ NeurIPS 2024

A Theory of Learning with Autoregressive Chain of Thought

Nirmit Joshi, Gal Vardi, Adam Block, Surbhi Goel, Zhiyuan Li, Theodor Misiakiewicz and Nathan Srebro

COLT 2025

Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context

Spencer Frei and Gal Vardi

ICLR 2025

Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks

Nikolaos Tsilivis, Eitan Gronich, Julia Kempe and Gal Vardi

ICLR 2025 (the above link is to an extended version)

A preliminary version appeared in M3L Workshop @ NeurIPS 2024

Approaching Deep Learning through the Spectral Dynamics of Weights

David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Vardi, Karen Livescu, Michael Maire and Matthew R. Walter

Preliminary versions appeared in HiLD and MI Workshops @ ICML 2024

Provable Tempered Overfitting of Minimal Nets and Typical Nets

Itamar Harel, William M. Hoza, Gal Vardi, Itay Evron, Nathan Srebro and Daniel Soudry

NeurIPS 2024

A preliminary version appeared in HiLD Workshop @ ICML 2024

Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality

Marko Medvedev, Gal Vardi and Nathan Srebro

NeurIPS 2024

Noisy Interpolation Learning with Shallow Univariate ReLU Networks

Nirmit Joshi, Gal Vardi and Nathan Srebro

ICLR 2024 (Selected as Spotlight)

An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression

Lijia Zhou, James B. Simon, Gal Vardi and Nathan Srebro

ICLR 2024

Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data

Zhiwei Xu, Yutong Wang, Spencer Frei, Gal Vardi and Wei Hu

ICLR 2024

A preliminary version appeared in M3L Workshop @ NeurIPS 2023 (Selected as Oral)

Most Neural Networks Are Almost Learnable

Amit Daniely*, Nathan Srebro* and Gal Vardi*

NeurIPS 2023

The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks

Spencer Frei*, Gal Vardi*, Peter L. Bartlett and Nathan Srebro

NeurIPS 2023

Computational Complexity of Learning Neural Networks: Smoothness and Degeneracy

Amit Daniely*, Nathan Srebro* and Gal Vardi*

NeurIPS 2023

Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Data Manifolds

Odelia Melamed, Gilad Yehudai and Gal Vardi

NeurIPS 2023

Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Losses

Gon Buzaglo*, Niv Haim*, Gilad Yehudai, Gal Vardi, Yakir Oz, Yaniv Nikankin and Michal Irani

NeurIPS 2023

A preliminary version appeared in TrustML-(un)Limited Workshop @ ICLR 2023

Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization

Spencer Frei*, Gal Vardi*, Peter L. Bartlett and Nathan Srebro

COLT 2023

On the Implicit Bias in Deep-Learning Algorithms (a survey)

Gal Vardi

Communications of the ACM (CACM), 2023

Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data

Spencer Frei*, Gal Vardi*, Peter L. Bartlett, Nathan Srebro and Wei Hu

ICLR 2023 (Selected as Spotlight)

Implicit Regularization Towards Rank Minimization in ReLU Networks

Nadav Timor, Gal Vardi and Ohad Shamir

ALT 2023

On Convexity and Linear Mode Connectivity in Neural Networks

David Yunis, Kumar Kshitij Patel, Pedro Savarese, Gal Vardi, Jonathan Frankle, Matthew Walter, Karen Livescu and Michael Maire

OPT Workshop @ NeurIPS 2022

On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias

Itay Safran*, Gal Vardi* and Jason D. Lee

NeurIPS 2022

Reconstructing Training Data from Trained Neural Networks

Niv Haim*, Gal Vardi*, Gilad Yehudai*, Ohad Shamir and Michal Irani

NeurIPS 2022 (Selected as Oral)

See project page

Gradient Methods Provably Converge to Non-Robust Networks

Gal Vardi*, Gilad Yehudai* and Ohad Shamir

NeurIPS 2022

The Sample Complexity of One-Hidden-Layer Neural Networks

Gal Vardi, Ohad Shamir and Nathan Srebro

NeurIPS 2022

On Margin Maximization in Linear and ReLU Networks

Gal Vardi, Ohad Shamir and Nathan Srebro

NeurIPS 2022

Width is Less Important than Depth in ReLU Neural Networks

Gal Vardi*, Gilad Yehudai* and Ohad Shamir

COLT 2022

On the Optimal Memorization Power of ReLU Neural Networks

Gal Vardi*, Gilad Yehudai* and Ohad Shamir

ICLR 2022 (Selected as Spotlight)

Learning a Single Neuron with Bias Using Gradient Descent

Gal Vardi*, Gilad Yehudai* and Ohad Shamir

NeurIPS 2021

From Local Pseudorandom Generators to Hardness of Learning

Amit Daniely* and Gal Vardi*

COLT 2021

Size and Depth Separation in Approximating Benign Functions with Neural Networks

Gal Vardi, Daniel Reichman, Toniann Pitassi and Ohad Shamir

COLT 2021

Implicit Regularization in ReLU Networks with the Square Loss

Gal Vardi and Ohad Shamir

COLT 2021

Neural Networks with Small Weights and Depth-Separation Barriers

Gal Vardi and Ohad Shamir

NeurIPS 2020

Hardness of Learning Neural Networks with Natural Weights

Amit Daniely* and Gal Vardi*

NeurIPS 2020

Perspective Games