Berivan Isik
I am a research scientist at Google, working on efficient and trustworthy AI.
My current interests are efficient training/finetuning of large models, pretraining data valuation and scaling laws for LLMs, differential privacy, and unlearning.
I completed my PhD at Stanford University, advised by Tsachy Weissman and Sanmi Koyejo, where I was affiliated with the SAIL and StatsML groups. My research was supported by Stanford Graduate Fellowship (2019-2023), Google Ph.D. Fellowship (2023-2026), and a Meta research grant.
If you have similar interests and would like to collaborate, please reach out via email:
berivan [at] google [dot] com
Publications
- Highlights:
Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings [PDF]
Pura Peetathawatchai, Wei-Ning Chen, Berivan Isik, Sanmi Koyejo, Albert No
Preprint, 2024.
Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs [PDF] [code]
Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, Prateek Mittal
Preprint, 2024.
Preliminary version accepted at the Workshop on Advancing Neural Network Training (WANT): Computational Efficiency, Scalability, and Resource Optimization, ICML, 2024. [Best Paper Award, Oral]
Preliminary version accepted at the Workshop on Efficient Systems for Foundation Models (ES-FoMo), ICML, 2024. [Oral]
Scaling Laws for Downstream Task Performance of Large Language Models [PDF]
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, Sanmi Koyejo
Preprint, 2024.
Preliminary version presented at the Workshop on Navigating and Addressing Data Problems for Foundation Models (DPFM), ICLR, 2024.
Preliminary version presented at the Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo), ICLR, 2024.
Sketching for Distributed Deep Learning: A Sharper Analysis [PDF]
Mayank Shrivastava, Berivan Isik, Qiabo Li, Arindam Banerjee, Sanmi Koyejo
Conference on Neural Information Processing Systems (NeurIPS), 2024.
Preliminary version accepted at the Workshop on Bridging the Gap Between Practice and Theory in Deep Learning, ICLR, 2024.
On Fairness of Low-Rank Adaptation of Large Models [PDF] [code]
Zhoujie Ding, Ken Liu, Pura Peetathawatchai, Berivan Isik, Sanmi Koyejo
Conference on Language Modeling (COLM), 2024.
Preliminary version presented at the Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo), ICLR, 2024.
Preliminary version presented at the Workshop on Secure and Trustworthy Large Language Models (SeT LLM), ICLR, 2024.
Improved Communication-Privacy Trade-offs in L2 Mean Estimation under Streaming Differential Privacy [PDF]
Wei-Ning Chen, Berivan Isik, Peter Kairouz, Albert No, Sewoong Oh, Zheng Xu
International Conference on Machine Learning (ICML), 2024.
Adaptive Compression in Federated Learning via Side Information [PDF] [code]
Berivan Isik*, Francesco Pase*, Deniz Gunduz, Sanmi Koyejo, Tsachy Weissman, Michele Zorzi
International Conference on Artificial Intelligence and Statistics (AISTATS), 2024.
Preliminary version presented at the Workshop on Federated Learning and Analytics, ICML, 2023.
Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimation [PDF] [code]
Berivan Isik, Wei-Ning Chen, Ayfer Ozgur, Tsachy Weissman, Albert No
Conference on Neural Information Processing Systems (NeurIPS), 2023.
Preliminary version presented at the ICML 2023 Federated Learning Workshop and the Theory and Practice of Differential Privacy (TPDP) Workshop in 2023.
Sparse Random Networks for Communication-Efficient Federated Learning [PDF] [code]
Berivan Isik*, Francesco Pase*, Deniz Gunduz, Tsachy Weissman, Michele Zorzi
International Conference on Learning Representations (ICLR), 2023.
Preliminary version presented at the Workshop on Federated Learning, NeurIPS, 2022. [Oral]
An Information-Theoretic Justification for Model Pruning [PDF] [code]
Berivan Isik, Tsachy Weissman, Albert No
International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
Preliminary version presented at the Sparsity in Neural Networks Workshop, 2021. [Spotlight talk]
- Others:
GPT-Zip: Deep Compression of Finetuned Large Language Models [PDF]
Berivan Isik*, Hermann Kumbong*, Wanyi Ning*, Xiaozhe Yao*, Sanmi Koyejo, Ce Zhang
Preliminary version presented at the Workshop on Efficient Systems for Foundation Models, ICML, 2023.
Sandwiched Compression: Repurposing Standard Codecs with Neural Network Wrappers [PDF][code]
Onur Guleryuz, Philip A. Chou, Berivan Isik, Hugues Hoppe, Danhang Tang, Ruofei Du, Jonathan Taylor, Philip Davidson, Sean Fanello
Submitted to the IEEE Transactions on Image Processing (TIP), 2024.
Mutual Information Upper Bounds for Uniform Inputs through the Deletion Channel [PDF]
Francisco Pernice, Berivan Isik, Tsachy Weissman
IEEE Transactions on Information Theory (TIT), 2024.
Lossy Compression of Noisy Data for Private and Data-Efficient Learning [PDF]
Berivan Isik, Tsachy Weissman
IEEE Journal of Selected Areas in Information Theory (JSAIT), 2023.
Sandwiched Video Compression: Efficiently Extending the Reach of Standard Codecs with Neural Wrappers [PDF] [code]
Berivan Isik, Onur Guleryuz, Danhang Tang, Jonathan Taylor, Philip A. Chou
IEEE International Conference on Image Processing (ICIP), 2023.
Neural Network Compression for Noisy Storage Devices [PDF]
Berivan Isik, Kristy Choi, Xin Zheng, Tsachy Weissman, Stefano Ermon, H.-S. Philip Wong, Armin Alaghi
ACM Transactions on Embedded Computing Systems (TECS), 2023.
Preliminary version presented at the Deep Learning through Information Geometry Workshop, NeurIPS, 2020.
Learning under Storage and Privacy Constraints [PDF1] [PDF2 (longer)]
Berivan Isik, Tsachy Weissman
IEEE International Symposium on Information Theory (ISIT), 2022.
LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks [PDF] [code]
Berivan Isik, Philip A. Chou, Sung Jin Hwang, Nick Johnston, George Toderici
Journal of Frontiers in Signal Processing (FSP), 2022.
Neural 3D Compression via Model Compression [PDF]
Berivan Isik
WiCV, CVPR, 2021.
rTop-k: A Statistical Estimation Approach to Distributed SGD [PDF]
Berivan Isik*, Leighton P. Barnes*, Huseyin A. Inan*, Ayfer Ozgur
IEEE Journal on Selected Areas in Information Theory (JSAIT), 2020.
Preliminary version presented at the Federated Learning for User Privacy and Data Confidentiality, ICML, 2020. [Long Oral]
PhD Thesis
Statistical methods for efficient and trustworthy machine learning (2024)
Patents
"Learned Volumetric Attribute Compression Using Coordinate-Based Networks", US Patent 17/708,628.
Philip A. Chou, Berivan Isik, Sung Jin Hwang, Nick Johnston, George Toderici
Invited Talks
University of Southern California - AI Foundations Seminar. "Value of Pretraining Data: Scaling Laws for Downstream Task Performance of Large Language Models", Oct 2024.
Vector Institute for Artificial Intelligence. "Scaling Laws for Downstream Task Performance of Large Language Models", March 2024.
INFORMS Optimization Society (IOS) Conference - Rice University. "Sparse Random Networks for Communication-Efficient Federated Learning", March 2024.
Columbia University - NYC Privacy Day. "Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimation", Dec. 2023.
Meta. "Federated Learning under Communication and Privacy Constraints", Oct. 2023.
Google - Federated Learning Seminar. "Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimation", Oct. 2023.
Vector Institute for Artificial Intelligence. "Sparse Random Networks for Communication-Efficient Federated Learning", Sep. 2023.
Video Quality Expert Group (VQEG). "Sandwiched Video Compression: Efficiently Extending the Reach of Standard Codecs with Neural Wrappers", June 2023.
Flower Monthly. "Sparse Random Networks for Communication-Efficient Federated Learning", June 2023.
Federated Learning One World (FLOW) Seminar. "Sparse Random Networks for Communication-Efficient Federated Learning", March 2023. [Slides]
Google - Federated Learning Seminar. "Sparse Random Networks for Communication-Efficient Federated Learning", Jan. 2023. [Slides]
Bilkent University - Electrical Engineering Seminars. "Sparsity in Neural Networks", Jan. 2023. [Slides]
Middle East Technical University - Electrical Engineering Seminars. "Sparsity in Neural Networks", Dec. 2022. [Slides]
Google - Sparsity Reading Group. "An Information-Theoretic Justification for Model Pruning", Nov. 2022.
Google - Perception. "Learned Sandwiched Video Compression", Oct. 2022.
Google - Information Theory Reading Group. "An Information-Theoretic Justification for Model Pruning", Sep. 2022.
Google - Information Theory Reading Group. "LCoN: Learning Under Storage and Privacy Constraints", July 2022.
Google - Neural Compression Team. "LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks", Sep. 2021.
Stanford Compression Workshop. "An Information-Theoretic Approach to Neural Network Compression", Feb. 2021. [Video] [Slides]
NeurIPS Meetup Turkey. "Noisy Neural Network Compression", Dec. 2020. [Video] (Turkish)
Facebook Reality Labs - Research. "Robust Neural Network Compression", Nov. 2020.
Involvements
Organizer of the ICLR 2025 Workshop on Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference.
Organizer of the ICML 2024 Workshop on Theoretical Foundations of Foundation Models (TF2M). [Link]
Organizer of the ICML 2024 Workshop on Data-centric Machine Learning Research (DMLR): Datasets for Foundation Models. [Link]
Organizer of the ICML 2023 Workshop on Neural Compression: From Information Theory to Applications. [Link]
Organizer of the Women in Machine Learning (WiML) Workshop at ICML 2021.
Organizer of the ICML 2021 Workshop on Information-Theoretic Methods for Rigorous, Responsible, and Reliable Machine Learning (ITR3@ICML-21). [Link]
Member of the Stanford Faculty Search Committee, 2024.
Member of the ICML 2024, NeurIPS 2024, and ICLR 2025 Workshop Proposal Committee.
Reviewer for:
International Conference on Machine Learning (ICML) 2021, 2022 (outstanding reviewer award), 2023, 2024
Conference on Neural Information Processing Systems (NeurIPS) 2021, 2022, 2023 (top reviewer)
International Conference on Learning Representations (ICLR) 2022, 2023, 2024, 2025
International Conference on Artificial Intelligence and Statistics (AISTATS) 2022, 2023, 2024, 2025
AAAI Conference on Artificial Intelligence (AAAI) 2025
Journal of Machine Learning Research (JMLR)
International Joint Conference on Artificial Intelligence (IJCAI) 2024
ACM Fairness, Accountability, and Transparency Conference (ACM FAccT) 2024
Journal of Data-centric Machine Learning Research (DMLR)
IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) 2024
European Conference on Computer Vision (ECCV) 2022-2024
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023-2024
International Conference on Computer Vision (ICCV) 2023
Data Compression Conference (DCC) 2024
SIAM Journal on Mathematics of Data Science (SIMODS)
IEEE Journal on Selected Areas in Information Theory (JSAIT)
IEEE International Symposium on Information Theory (ISIT) 2022-2024
IEEE Information Theory Workshop (ITW) 2022
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
Reviewer for the ICLR Blogposts track, 2023-2024.
Program committee member for:
ECCV 2024 Unlearning and Model Editing Workshop
KDD 2024 Workshop on Federated Learning for Data Mining and Graph Analytics
ICML 2024 Workshop on Efficient Systems for Foundation Models
ICML 2024 Workshop on Models of Human Feedback for AI Alignment
ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models
ICLR 2024 Workshop on Privacy Regulation and Protection in Machine Learning
NeurIPS 2023 Workshop, Self-Supervised Learning - Theory and Practice
NeurIPS 2023 Workshop, UniReps: Unifying Representations in Neural Models
NeurIPS 2023 Workshop, R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Foundation Models
NeurIPS 2023 Workshop on Federated Learning in the Age of Foundation Models
ICML 2023 Workshop on New Frontiers in Adversarial Machine Learning
ICML 2023 Workshop on Knowledge and Logical Reasoning in the Era of Data-Driven Learning
ICML 2023 Workshop on Federated Learning and Analytics in Practice
NeurIPS 2022 Workshop on Federated Learning: Recent Advances and New Challenges
Mentor for STEM to SHTEM summer internship program for high school students in 2020.
Hobbies
I like photography and reading.