Hello! I am Research Scientist at NVIDIA and concentrating on building high-performing, efficient-size vision language models by merging proper modules, modifying architectures, and distilling cross-model or self-model knowledge, thereby surpassing GPT-4V! Main research interests are distillation, pruning, compression as well as potentially data/training/inference efficiency.
I am seeking highly motivated Ph.D. students for NVIDIA Research Interns, based on research interest below. Also, welcome to research collaborations. Feel free to contact me by mail.
Distillation
Pruning
Compression
Data/Training/Inference Efficiency (Potentially)
Ph.D: School of Electrical Engineering, KAIST, Korea (Feb. 2020 ~ Aug. 2025)
"Dissertation: Building High-performing, Efficient Size Vision Language Models: Merge, Modify, and Distill [link]"
M.S: The CCS graduate school of Mobility, KAIST, Korea (Mar. 2018 ~ Feb. 2020)
"Thesis: Training encoder-attention through fully-connected CRFs for efficient end-to-end lane detection model [link]"
B.S: Department of Mathematics (Electronic Engineering), Hanyang University, Korea (Mar. 2014 ~ Feb. 2018)
Research Scientist
Oct. 2025 ~ Current
Research Intern
Oct. 2024 ~ Oct. 2025
Publications (13 Accepts / 2 Tech Reports / 4 Pending)
CV: 3 CVPR, 2 ICCV, 1 ECCV, 1 ICIP
ML: 3 NeurIPS
NLP: 1 ACL, 1 EMNLP
Journal: 1 Pattern Recognition
"GenRecal: Generation after Recalibration from Large to Small Vision Language Models"
Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro, Yu-Chiang Frank Wang, and Yueh-Hua Wu
-CVPR 2026 Submission [link][project]
-Featured by 🤗 Huggingface Daily Papers
"Recursive Think-Answer Process for LLMs and VLMs"
Byung-Kwan Lee, Youngchae Chee, Yong Man Ro
-CVPR 2026 Submission [link][project]
"RefineBench: Evaluating Refinement Capability in Language Models"
Young-Jun Lee*, Seungone Kim*, Byung-Kwan Lee, Minkyeong Moon, Yechan Hwang, Jong Myoung Kim, Graham Neubig, Sean Welleck, Ho-Jin Choi
-ICLR 2026 Submission [link][project]
-Workshop for Multi-Turn Interactions in Large Language Models (Oral) [link]
"Enhancing Conversational Agents with Skill-of-Mind-Infused Large Language Model"
Young-Jun Lee, Byung-Kwan Lee, Dokyong Lee, junyoung youn, Kyeong-Jin Oh, Yechan Hwang, Ho-Jin Choi
-ICLR 2026 Submission [link][project]
"Unified Reinforcement and Imitation Learning for Vision-Language Models"
Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro, Yu-Chiang Frank Wang, Yueh-Hua Wu
-Neural Information Processing Systems (NeurIPS), 2025 [link][project]
"MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models"
Young-Jun Lee, Byung-Kwan Lee, Jianshu Zhang, Yechan Hwang, Byungsoo Ko, Han-Gyu Kim, Dongyu Yao, Xuankun Rong, Eojin Joo, Seung-Ho Han, Bowon Ko, Ho-Jin Choi
-IEEE/CVF International Conference on Computer Vision (ICCV), 2025 [link][project]
-Workshop for Knowledge-Intensive Multimodal Reasoning [link]
"VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models"
Byung-Kwan Lee, Ryo Hachiuma, Yu-Chiang Frank Wang, Yong Man Ro, Yueh-Hua Wu
-IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025 [link][project]
-Featured by 🤗 Huggingface Daily Papers
"Phantom of Latent for Large Language and Vision Models"
Byung-Kwan Lee, Sangyun Chung, Chae Won Kim, Beomchan Park, Yong Man Ro
-ArXiv 2024, Technical Report [link][code]
-Featured by 🤗 Huggingface Daily Papers
"TroL: Traversal of Layers for Large Language and Vision Models"
Byung-Kwan Lee, Sangyun Chung, Chae Won Kim, Beomchan Park, Yong Man Ro
-Empirical Methods in Natural Language Processing (EMNLP), 2024 [link][code]
-Featured by 🤗 Huggingface Daily Papers
"Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models"
Byung-Kwan Lee, Chae Won Kim, Beomchan Park, Yong Man Ro
-Neural Information Processing Systems (NeurIPS), 2024 [link][code]
-Featured by 🤗 Huggingface Daily Papers
-31th Samsung HumanTech Paper Awards in section of Computer Science & Engineering
"MoAI: Mixture of All Intelligence for Large Language and Vision Models"
Byung-Kwan Lee, Beomchan Park, Chae Won Kim, Yong Man Ro
-European Conference on Computer Vision (ECCV), 2024 [link][code]
-Featured by 🤗 Huggingface Daily Papers
"CoLLaVO: Crayon Large Language and Vision mOdel"
Byung-Kwan Lee, Beomchan Park, Chae Won Kim, Yong Man Ro
-Findings of the Association for Computational Linguistics (ACL), 2024 [link][code]
-Featured by 🤗 Huggingface Daily Papers
-2024 KCC XAI Workshop, Best Paper Awards
"Causal Unsupervised Semantic Segmentation"
Junho Kim*, Byung-Kwan Lee*, and Yong Man Ro
-Patten Recognition [link][code]
"Mitigating Adversarial Vulnerability through Causal Parameter Estimation by Adversarial Double Machine Learning"
Byung-Kwan Lee*, Junho Kim*, and Yong Man Ro
-IEEE/CVF International Conference on Computer Vision (ICCV), 2023 [link][code]
"Mitigating Dataset Bias in Image Captioning through CLIP Confounder-free Captioning Network"
YeonJu Kim, Junho Kim, Byung-Kwan Lee, Sebin Shin, and Yong Man Ro
-IEEE International Conference on Image Processing (ICIP), 2023 [link][code]
"Demystifying Causal Features on Adversarial Examples and Causal Inoculation for Robust Network by Adversarial Instrumental Variable Regression"
Junho Kim*, Byung-Kwan Lee*, and Yong Man Ro
-IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023 [link][code]
"Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Network"
Byung-Kwan Lee*, Junho Kim*, and Yong Man Ro
-IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022 [link][code]
"Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck"
Junho Kim*, Byung-Kwan Lee*, and Yong Man Ro
-Neural Information Processing Systems (NeurIPS), 2021 [link][code]
"Towards Adversarial Robustness of Bayesian Neural Network through Hierarchical Variational Inference"
Byung-Kwan Lee, Youngjoon Yu, Yong Man Ro
Journal
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
IEEE Transactions on Neural Networks and Learning Systems (TNNLS)
Elsevier on Computer Vision and Image Understanding (CVIU)
Conference
Computer Vision and Pattern Recognition (CVPR) 2021/2022/2023/2024
International Conference on Computer Vision (ICCV) 2021/2023/2025
European Conference on Computer Vision (ECCV) 2022/2024
International Conference on Learning Representations (ICLR) 2023/2024/2025
Neural Information Processing Systems (NeurIPS) 2023/2024/2025
International Conference on Machine Learning (ICML) 2024/2025
Association for the Advancement of Artificial Intelligence (AAAI) Program Committee 2025
Asian Conference on Computer Vision (ACCV) 2024
Talk and Award
Kookmin University Invited Talk, Building High-performing, Efficient-size Vision Language Models, 2025
Dongseo University Invited Talk, Building High-performing, Efficient-size Vision Language Models, 2025
Korea Institute of Science and Technology Information (KISTI) Invited Talk, Efficient Large Language and Vision Models, 2024
NAVER HyperCLOVA X Invited Talk, Efficient Large Language and Vision Models, 2024
Kakao Brain Talk Invited Talk, Efficient Large Language and Vision Models, 2024
Center for Applied Research in Artificial Intelligence (CARAI) Paper Seminar, 1st Award for Research Presentation, 2023
GIRE Research Mentor for Seoul Interantional School - "MUsE: Model of User-friendliness and Efficiency", Journal of Student Research, Publication [link]
Korea Advanced Institute of Science and Technology (KAIST) SW IT Academy - AI Project Director of 1st Award Topic: ``Speech to Storybook Generation for Children using Finetuning OpenAI's Whisper, GPT-4, and Stable Diffusion'', 2023
Invitation to Presenter in Korean Conference on Computer Vision (KCCV), 2023
Korea Advanced Institute of Science and Technology (KAIST) SW IT Academy, Lecturer of Data Structure & Algorithm, Deep Learning, Computer Vision, 2023
Invitation to Presenter in Korean Conference on Computer Vision (KCCV), 2022
Center for Applied Research in Artificial Intelligence (CARAI) Paper Seminar, 2nd Award for Research Presentation, 2022
Korea Advanced Institute of Science and Technology (KAIST) Fellowship, 2020 - Present
National Government Fellowship 2018 - 2020
HYUNDAI Autonomous Vehicle Competition 2019 held in South Korea, Team Leader, 2018-2019
President's Award for International Autonomous Vehicle Competition 2018 held in South Korea, Team Leader, [YouTube][Talk]