Berivan Isik
I am a final year PhD student at Stanford University advised by Tsachy Weissman and Sanmi Koyejo, where I am affiliated with the SAIL and StatsML groups. My research is supported by Stanford Graduate Fellowship (2019-2023), Google Ph.D. Fellowship (2023-2026), and a Meta research grant.
I previously interned at Google (2021, 2022, 2023-2024), Amazon (2022-2023), and Vector Institute for AI (2023).
My research focuses on efficient & trustworthy machine learning. Recently, I have been working on differential privacy, model compression, federated learning, transfer learning, data valuation and scaling laws in large language models, and efficient training/finetuning of large language models.
If you have similar interests and would like to collaborate, please reach out via email:
berivan.isik [at] stanford [dot] edu
Publications
- Highlights:
Scaling Laws for Downstream Task Performance of Large Language Models [PDF]
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, Sanmi Koyejo
Preprint, 2024.
Preliminary version accepted at the Workshop on Navigating and Addressing Data Problems for Foundation Models (DPFM), ICLR, 2024.
Preliminary version accepted at the Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo), ICLR, 2024.
On Fairness Implications and Evaluations of Low-Rank Adaptation of Large Models
Ken Liu, Zhoujie Ding, Berivan Isik, Sanmi Koyejo
Preprint, 2024.
Preliminary version accepted at the Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo), ICLR, 2024.
Preliminary version accepted at the Workshop on Secure and Trustworthy Large Language Models (SeT LLM), ICLR, 2024.
Improved Communication-Privacy Trade-offs in L2 Mean Estimation under Streaming Differential Privacy
Wei-Ning Chen, Berivan Isik, Peter Kairouz, Albert No, Sewoong Oh, Zheng Xu
International Conference on Machine Learning (ICML), 2024.
Adaptive Compression in Federated Learning via Side Information [PDF] [code]
Berivan Isik*, Francesco Pase*, Deniz Gunduz, Sanmi Koyejo, Tsachy Weissman, Michele Zorzi
International Conference on Artificial Intelligence and Statistics (AISTATS), 2024.
Preliminary version presented at the Workshop on Federated Learning and Analytics, ICML, 2023.
Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimation [PDF] [code]
Berivan Isik, Wei-Ning Chen, Ayfer Ozgur, Tsachy Weissman, Albert No
Conference on Neural Information Processing Systems (NeurIPS), 2023.
Preliminary version presented at the ICML 2023 Federated Learning Workshop and the Theory and Practice of Differential Privacy (TPDP) Workshop in 2023.
Sparse Random Networks for Communication-Efficient Federated Learning [PDF] [code]
Berivan Isik*, Francesco Pase*, Deniz Gunduz, Tsachy Weissman, Michele Zorzi
International Conference on Learning Representations (ICLR), 2023.
Preliminary version presented at the Workshop on Federated Learning, NeurIPS, 2022. [Oral]
An Information-Theoretic Justification for Model Pruning [PDF] [code]
Berivan Isik, Tsachy Weissman, Albert No
International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
Preliminary version presented at the Sparsity in Neural Networks Workshop, 2021. [Spotlight talk]
- Others:
Breaking the Dimension Dependence in Sketching for Distributed Learning
Berivan Isik*, Mayank Shrivastava*, Qiabo Li, Arindam Banerjee, Sanmi Koyejo
Preprint, 2024.
Preliminary version accepted at the Workshop on Bridging the Gap Between Practice and Theory in Deep Learning, ICLR, 2024.
GPT-Zip: Deep Compression of Finetuned Large Language Models [PDF]
Berivan Isik*, Hermann Kumbong*, Wanyi Ning*, Xiaozhe Yao*, Sanmi Koyejo, Ce Zhang
Preliminary version presented at the Workshop on Efficient Systems for Foundation Models, ICML, 2023.
Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations [PDF]
Rylan Schaeffer, Berivan Isik, et.al.
Preprint, 2024.
Preliminary version accepted at the Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo), ICLR, 2024.
Preliminary version presented at the Workshop on Unifying Representations in Neural Models (UniReps), NeurIPS, 2023. [Oral, Outstanding Paper Award]
Preliminary version presented at the Workshop on Information-Theoretic Principles in Cognitive Systems (InfoCog), NeurIPS, 2023. [Spotlight talk]
Sandwiched Compression: Repurposing Standard Codecs with Neural Network Wrappers [paper][code]
Onur Guleryuz, Philip A. Chou, Berivan Isik, Hugues Hoppe, Danhang Tang, Ruofei Du, Jonathan Taylor, Philip Davidson, Sean Fanello
Submitted to the IEEE Transactions on Image Processing (TIP), 2024.
Mutual Information Upper Bounds for Uniform Inputs through the Deletion Channel [PDF]
Francisco Pernice, Berivan Isik, Tsachy Weissman
IEEE Transactions on Information Theory (TIT), 2024.
Lossy Compression of Noisy Data for Private and Data-Efficient Learning [PDF]
Berivan Isik, Tsachy Weissman
IEEE Journal of Selected Areas in Information Theory (JSAIT), 2023.
Sandwiched Video Compression: Efficiently Extending the Reach of Standard Codecs with Neural Wrappers [PDF] [code]
Berivan Isik, Onur Guleryuz, Danhang Tang, Jonathan Taylor, Philip A. Chou
IEEE International Conference on Image Processing (ICIP), 2023.
Neural Network Compression for Noisy Storage Devices [PDF]
Berivan Isik, Kristy Choi, Xin Zheng, Tsachy Weissman, Stefano Ermon, H.-S. Philip Wong, Armin Alaghi
ACM Transactions on Embedded Computing Systems (TECS), 2023.
Preliminary version presented at the Deep Learning through Information Geometry Workshop, NeurIPS, 2020.
Learning under Storage and Privacy Constraints [PDF1] [PDF2 (longer)]
Berivan Isik, Tsachy Weissman
IEEE International Symposium on Information Theory (ISIT), 2022.
LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks [PDF] [code]
Berivan Isik, Philip A. Chou, Sung Jin Hwang, Nick Johnston, George Toderici
Journal of Frontiers in Signal Processing (FSP), 2022.
Neural 3D Compression via Model Compression [PDF]
Berivan Isik
WiCV, CVPR, 2021.
rTop-k: A Statistical Estimation Approach to Distributed SGD [PDF]
Berivan Isik*, Leighton P. Barnes*, Huseyin A. Inan*, Ayfer Ozgur
IEEE Journal on Selected Areas in Information Theory (JSAIT), 2020.
Preliminary version presented at the Federated Learning for User Privacy and Data Confidentiality, ICML, 2020. [Long Oral]
Patents
"Learned Volumetric Attribute Compression Using Coordinate-Based Networks", US Patent 17/708,628.
Philip A. Chou, Berivan Isik, Sung Jin Hwang, Nick Johnston, George Toderici
Invited Talks
Vector Institute for Artificial Intelligence - "Scaling Laws for Downstream Task Performance of Large Language Models", March 2024.
INFORMS Optimization Society (IOS) Conference - Rice University. "Sparse Random Networks for Communication-Efficient Federated Learning", March 2024.
Columbia University - NYC Privacy Day. "Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimation", Dec. 2023.
Meta. "Federated Learning under Communication and Privacy Constraints", Oct. 2023.
Google - Federated Learning Seminar. "Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimation", Oct. 2023.
Vector Institute for Artificial Intelligence. "Sparse Random Networks for Communication-Efficient Federated Learning", Sep. 2023.
Video Quality Expert Group (VQEG). "Sandwiched Video Compression: Efficiently Extending the Reach of Standard Codecs with Neural Wrappers", June 2023.
Flower Monthly. "Sparse Random Networks for Communication-Efficient Federated Learning", June 2023.
Federated Learning One World (FLOW) Seminar. "Sparse Random Networks for Communication-Efficient Federated Learning", March 2023. [Slides]
Google - Federated Learning Seminar. "Sparse Random Networks for Communication-Efficient Federated Learning", Jan. 2023. [Slides]
Bilkent University - Electrical Engineering Seminars. "Sparsity in Neural Networks", Jan. 2023. [Slides]
Middle East Technical University - Electrical Engineering Seminars. "Sparsity in Neural Networks", Dec. 2022. [Slides]
Google - Sparsity Reading Group. "An Information-Theoretic Justification for Model Pruning", Nov. 2022.
Google - Perception. "Learned Sandwiched Video Compression", Oct. 2022.
Google - Information Theory Reading Group. "An Information-Theoretic Justification for Model Pruning", Sep. 2022.
Google - Information Theory Reading Group. "LCoN: Learning Under Storage and Privacy Constraints", July 2022.
Google - Neural Compression Team. "LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks", Sep. 2021.
Stanford Compression Workshop. "An Information-Theoretic Approach to Neural Network Compression", Feb. 2021. [Video] [Slides]
NeurIPS Meetup Turkey. "Noisy Neural Network Compression", Dec. 2020. [Video] (Turkish)
Facebook Reality Labs - Research. "Robust Neural Network Compression", Nov. 2020.
Involvements
Organizer of the ICML 2024 Workshop on Theoretical Foundations of Foundation Models (TF2M). [Link]
Organizer of the ICML 2024 Workshop on Data-centric Machine Learning Research (DMLR): Datasets for Foundation Models.
Organizer of the ICML 2023 Workshop on Neural Compression: From Information Theory to Applications. [Link]
Organizer of the Women in Machine Learning (WiML) Workshop at ICML 2021.
Organizer of the ICML 2021 Workshop on Information-Theoretic Methods for Rigorous, Responsible, and Reliable Machine Learning (ITR3@ICML-21). [Link]
Member of the Stanford Faculty Search Committee, 2024.
Member of the ICML 2024 Workshop Proposal Committee.
Reviewer for:
International Conference on Machine Learning (ICML) 2021, 2022 (outstanding reviewer award), 2023, 2024
Conference on Neural Information Processing Systems (NeurIPS) 2021, 2022, 2023 (top reviewer)
International Conference on Learning Representations (ICLR) 2022, 2023, 2024
International Conference on Artificial Intelligence and Statistics (AISTATS) 2022, 2023, 2024
Journal of Machine Learning Research (JMLR)
International Joint Conference on Artificial Intelligence (IJCAI) 2024
ACM Fairness, Accountability, and Transparency Conference (ACM FAccT) 2024
Journal of Data-centric Machine Learning Research (DMLR)
IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) 2024
European Conference on Computer Vision (ECCV) 2022-2024
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023-2024
International Conference on Computer Vision (ICCV) 2023
Data Compression Conference (DCC) 2024
SIAM Journal on Mathematics of Data Science (SIMODS)
IEEE Journal on Selected Areas in Information Theory (JSAIT)
IEEE International Symposium on Information Theory (ISIT) 2022-2024
IEEE Information Theory Workshop (ITW) 2022
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
Reviewer for the ICLR Blogposts track, 2023-2024.
Program committee member for:
ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models
ICLR 2024 Workshop on Privacy Regulation and Protection in Machine Learning
NeurIPS 2023 Workshop, Self-Supervised Learning - Theory and Practice
NeurIPS 2023 Workshop, UniReps: Unifying Representations in Neural Models
NeurIPS 2023 Workshop, R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Foundation Models
NeurIPS 2023 Workshop on Federated Learning in the Age of Foundation Models
ICML 2023 Workshop on New Frontiers in Adversarial Machine Learning
ICML 2023 Workshop on Knowledge and Logical Reasoning in the Era of Data-Driven Learning
ICML 2023 Workshop on Federated Learning and Analytics in Practice
NeurIPS 2022 Workshop on Federated Learning: Recent Advances and New Challenges
Grader for the course EE 382A: Parallel Processors Beyond Multicore Processing in Spring 2021.
Mentor for STEM to SHTEM summer internship program for high school students in 2020.
Hobbies
Photography: my amateur photographs