Jongwoo Ko
I am a Ph.D. student of the KAIST AI and a member of OSI Lab (Advisor: Se-young Yun, KAIST). My current research focuses on efficient Transformer models [C6, C7, C8, C10], particularly generative language models like T5 or LLaMA. Additionally, I am also interested in efficient Vision Transformer models or multi-modal models [P1]. I aim to enhance the efficiency of large Transformer models.
Previously, my research interests revolved around developing new algorithms to address real-world challenges in the machine learning pipeline, such as noise label [C1, C3, W4] and class imbalance [C5] settings, while providing statistical or mathematical guarantees. I received a master's degree in the Department of Industrial and Systems Engineering from KAIST under the supervision of Prof. Heeyoung Kim.
Contact me : jongwoo [dot] ko [at] kaist [dot] ac [dot] kr [CV / Scholar / Github / LinkedIn]
Publications 📑
(J: Journal, C: Conference, W: Workshop, *: Equal Contribution, ^: Equal Advising)
2024
[C9] Fine-tuning Pre-trained Models for Robustness Under Noisy Labels
Sumyeong Ahn, Sihyeon Kim, Jongwoo Ko, Se-Young Yun.
International Joint Conference on Artificial Intelligence (IJCAI). 2024. Jeju [paper]
2023
[C8] NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models
[C7] Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding
[C6] Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective
[C5/W2/W3] CUDA: Curriculum of Data Augmentation for Long-tailed Recognition
Sumyeong Ahn*, Jongwoo Ko*, Se-Young Yun
[C5] The Eleventh International Conference on Learning Representations (ICLR). 2023. Kigali. (Notable-Top-25%) [paper] [code]
[W2] NeurIPS 2022 Workshop on Distribution Shifts (DistShift). 2022. New Orleans [paper] [website]
[W3] NeurIPS 2022 ML Safety Workshop (MLSW). 2022. New Orleans [paper] [website]
[C3/W1] A Gift from Label Smoothing: Robust Training with Adaptive Label Smoothing via Auxiliary Classifier under Label Noise
2022
[J2] Deep Learning-Based Cataract Detection and Grading from Slit-Lamp and Retro-Illumination Photographs: Model Development and Validation Study
Ki Young Son*, Jongwoo Ko*, Eunseok Kim, Si Young Lee, Min-Ji Kim, Jisang Han, Eunhae Shin, Tae-Young Chung, Dong Hui Lim
Ophthalmology Science. 2(2). 100147 [paper]
2021
[C1] FINE Samples for Learning with Noisy Labels
Experience 🌏
Applied Scientist Intern @Amazon.com Services LLC
Sunnyvale, California, United States
Apr. 2024 - Jun. 2024 (12 Weeks)
Invited Talk 📢
ASG (Applied Science Group) Research Talk @Microsoft
Title: Efficient Knowledge Distillation for sLLMs & On-device Generative AI Models
Introducing my recent paper "DistiLLM: Towards Streamlined Distillation for Large Language Models"
Host: Tianyi Chen
Code Implementations 🖥️
Pytorch-MiniLM
Unofficial Pytorch Reimplementation of MiniLM and MiniLMv2.
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (NeurIPS 2020)
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (Findings of ACL 2021)
Awards & Honors 🏆
Silver Prize, 30th Samsung Humantech Paper Awards (2024)
Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding
Winner, Qualcomm Innovation Fellowship Korea (2022)
FINE Samples for Learning with Noisy Labels
Editor's Choice for Featured Article, IISE Transactions (2022)
Deep Gaussian Process Models for Integrating Multifidelity Experiments with Non-stationary Relationships
Education 🧑🎓
Korea Advanced Institue of Science and Technology (KAIST), Seoul, Korea, Mar. 2020 - Present
Ph.D in Kim Jaechul Graduate School of Artificial Intelligence (Advisor: Se-Young Yun)
Korea Advanced Institue of Science and Technology (KAIST), Daejeon, Korea, Mar. 2018 - Feb. 2020
M.S. in Department of Industrial and Systems Engineering (Advisor: Heeyoung Kim)
Thesis: Deep Gaussian Process Models for Integrating Multifidelity Experiments with Non-stationary Relationships
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea, Mar. 2014 - Feb. 2018
B.S. in Department of Industrial and Systems Engineering (Magna Cum Laude)