Yanghua Peng

Email: whucspyh AT gmail.com

I am a research scientist and engineering lead at Seed, ByteDance, based in Seattle. I earned my Ph.D. in Computer Science from the University of Hong Kong, where I was advised by Prof. Chuan Wu, and I hold a bachelor's degree from Wuhan University.

My work focuses on optimizing the performance of large-scale training for foundation LLMs, multi-modal LLMs and video generation models. At ByteDance, we are developing cutting-edge AI infrastructure, including AI system co-design, computation and communication optimizations, 5D parallelisms, fast checkpointing and data IO, fault tolerance, and more.

My research interests span machine learning systems and cluster scheduling. Feel free to explore our recent system projects, such as ByteCheckpoint, VeRL and MegaScale, as well as our latest product updates with Doubao.

We are hiring! Contact me if you are interested in joining our team.

Publications

Machine Learning Systems

[Preprint'25] MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production

Chao Jin, Ziheng Jiang, Zhihao Bai, Zheng Zhong, Juncai Liu, Xiang Li, Ningxin Zheng, Xi Wang, Cong Xie, Wen Heng, Yiyuan Ma, Wenlei Bao, Size Zheng, Yanghua Peng, Haibin Lin, Xuanzhe Liu, Xin Jin, Xin Liu

[ATC'25] Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation

Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng, Haibin Lin, Minlan Yu

[Preprint'25] OVERLORD: Ultimate Scaling of DataLoader for Multi-Source Large Foundation Model Training

Juntao Zhao, Qi Lu, Wei Jia, Borui Wan, Lei Zuo, Junda Feng, Jianyu Jiang, Yangrui Chen, Shuaishuai Cao, Jialing He, Kaihua Jiang, Yuanzhe Hu, Yanghua Peng, Haibin Lin, Xin Liu, Chuan Wu

[NSDI'25] ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development

Borui Wan, Mingji Han, Yiyao Sheng, Yanghua Peng, Haibin Lin, Mofan Zhang, Zhichao Lai, Menghan Yu, Junda Zhang, Zuquan Song, Xin Liu, Chuan Wu

[EuroSys'25] HybridFlow/veRL: A Flexible and Efficient RLHF Framework

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, Chuan Wu

GitHub verl project

[NSDI'24] MegaScale: Scaling Large Language Modeling Training to More Than 10,000 GPUs

Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, etc.

GitHub vescale project

[EuroSys'24] CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs

Hanpeng Hu, Junwei Su, Juntao Zhao, Yanghua Peng, Yibo Zhu, Haibin Lin, Chuan Wu

[IPDPS'24] QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices

Juntao Zhao, Borui Wan, Yanghua Peng, Haibin Lin, Yibo Zhu, Chuan Wu

[PPoPP'24] LLM-PQ:Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization

Juntao Zhao, Borui Wan, Chuan Wu, Yanghua Peng, Haibin Lin

[NSDI'23] BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing

Tianfeng Liu, Yangrui Chen, Dan Li, Chuan Wu, Yibo Zhu, Jun He, Yanghua Peng, Hongzheng Chen, Hongzhi Chen, Chuanxiong Guo

[Neural Networks'23] SP-GNN: Learning Structure and Position Information from Graphs

Yangrui Chen, Jiaxuan You, Jun He, Yuan Lin, Yanghua Peng, Chuan Wu, Yibo Zhu

[NeurIPS'22] SAPipe: Staleness-Aware Pipeline for Data Parallel DNN Training

Yangrui Chen, Cong Xie, Meng Ma, Juncheng Gu, Yanghua Peng, Haibin Lin, Chuan Wu, Yibo Zhu

[MLSys'22] dPRO: A Generic Performance Diagnosis and Optimization Toolkit for Expediting Distributed DNN Training

Hanpeng Hu, Chenyu Jiang, Yuchen Zhong, Yanghua Peng, Chuan Wu, Yibo Zhu, Haibin Lin

[Communications of the CCF'21] 分布式深度学习训练的通信加速

Yimin Jiang, Yanghua Peng, Yibo Zhu, Chuanxiong Guo

[SoCC'20] Elastic Parameter Server Load Distribution in Deep Learning Clusters

Yangrui Chen, Yanghua Peng, Yixin Bao, Chuan Wu, Yibo Zhu, Chuanxiong Guo

[INFOCOM'20] Preemptive All-reduce Scheduling for Expediting Distributed DNN Training

Yixin Bao, Yanghua Peng, Yangrui Chen, Chuan Wu

[SOSP'19] A Generic Communication Scheduler for Distributed DNN Training Acceleration

Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, Chuanxiong Guo

GPU Cluster Scheduling

[SIGCOMM'22] Multi-Resource Interleaving for Deep Learning Training

Yihao Zhao, Yuanqiang Liu, Yanghua Peng, Yibo Zhu, Xuanzhe Liu, Xin Jin

[TON'22] Deep Learning-Based Job Placement in Distributed Machine Learning Clusters With Heterogeneous Workloads

Yixin Bao, Yanghua Peng, Chuan Wu

[TPDS'21] DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters

Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, Chen Meng, Wei Lin

[INFOCOM'19] Deep Learning-based Job Placement in Distributed Machine Learning Clusters

Yixin Bao, Yanghua Peng, Chuan Wu

[EuroSys'18] Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters

Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, Chuanxiong Guo

[INFOCOM'18] Online Job Scheduling in Distributed Machine Learning Clusters

Yixin Bao, Yanghua Peng, Yangrui Chen, Chuan Wu

Networks

[JSAC'18] Dynamic Scaling of Virtualized, Distributed Service Chains: A Case Study of IMS

Jingpu Duan, Chuan Wu, Franck Le, Alex X. Liu, Yanghua Peng

[ATC'18] deTector: a Topology-aware Monitoring System for Data Center Networks

Yanghua Peng, Ji Yang, Chuan Wu, Chuanxiong Guo, Chengchen Hu, Zongpeng Li

Awards

AliStar, Huawei Top Minds, SenseTime AI Pioneer 2019
SOSP Student Scholarship 2019
Lee Shau Kee Postgraduate Fellowship 2016
OutStanding Undergraduate Scholarship 2015
Google Excellence Scholarship 2014
National First Prize in Mathematical Contest in Modeling 2013
First Prize of New Enrollment Scholarship 2011

Page updated

Google Sites

Report abuse