Hyemin Jeong
Hyemin Jeong
Embedded Files
Welcome to my website! 😊
I’m Hyemin Jeong, a master’s student at the Graduate School of Data Science, Seoul National University. I’m currently conducting research under the supervision of Prof. Joonseok Lee at Visual Information Processing Lab (VIP). My research interests include Multimodal Large Language Models (MLLMs), cross-modal alignment, and efficient model design.
Feel free to contact me through the links below. Here is my CV.
Education
Education
Publications
Publications
* indicates equal contribution
* indicates equal contribution
TripleSumm: Adaptive Triple-Modality Fusion for Video SummarizationSumin Kim*, Hyemin Jeong*, Mingu Kang*, Yejin Kim, Yoori Oh, Joonseok LeeProceedings of the 14th International Conference on Learning Representations (ICLR), 2026[website] [paper] [github] [huggingface]
- Most video summarization models apply fixed fusion strategies across all frames, but in practice, the most informative modality shifts moment to moment.
Can a model learn when to trust each modality, rather than always averaging them? - We designed an adaptive trimodal fusion framework that dynamically reweights visual, audio, and textual signals based on frame-level saliency, achieving 1.37M parameters (87% reduction), 0.97 GFLOPs, and 2.81ms inference latency. To support rigorous evaluation, we also curated MoSu — a ~52K-video, 4,000-hour benchmark with synchronized multimodal annotations.
A More Word-like Image Tokenization for MLLMsHyun Lee, Hyemin Jeong, Yejin Kim, Hyungwook Choi, Hyunsoo Cho, Soo Kyung Kim, Joonseok LeeIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2026[paper]
- Visual tokens in multimodal LLMs are typically dense grid patches that flood the model with redundant low-level information, which is a stark contrast to how language encodes meaning compactly in words.
- Taking inspiration from linguistic tokenization, we group patch embeddings into semantically coherent visual tokens with adaptive budget allocation, reducing token count by up to 93.8% while maintaining strong multimodal reasoning performance.
K/DA: Automated Data Generation Pipeline for Detoxifying Implicitly Offensive Language in Korean Minkyeong Jeon*, Hyemin Jeong*, Yerang Kim, Jiyoung Kim, JaeHyeon Cho, Byung-Jun Lee Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025[website] [paper] [github] [huggingface]
- Implicit offensive language — sarcasm, coded insults, culturally loaded expressions — is far harder to detoxify than explicit slurs, yet paired datasets for Korean are nearly nonexistent.
- We built a RAG-based pipeline embedding a ~93K-sentence vector database to automatically synthesize 7,555 toxic–neutral pairs, then instruction fine-tuned multiple LLMs (LLaMA3-8B, Gemma2-9B), achieving 86% pair consistency and 90% preservation of implicit toxicity signals.
Patents
Patents
Method and System for Detoxifying Offensive Language using AI ModelsByung-Jun Lee, Hyemin Jeong, Minkyeong Jeon, Yerang Kim, and Jiyoung KimKorean Patent Application No. 10-2024-0174566, filed Nov 2024 (Applicant: Korea University Research \& Business Foundation)
Method and Apparatus for Adaptive Triple-Modality Fusion for Video SummarizationMingu Kang, Sumin Kim, Yejin Kim, Yoori Oh, Hyemin Jeong, and Joonseok LeeKorean patent, specification under drafting
Method and Apparatus for Adaptive Triple-Modality Fusion for Video SummarizationMingu Kang, Sumin Kim, Yejin Kim, Yoori Oh, Hyemin Jeong, and Joonseok LeeKorean patent, specification under drafting
Grants & Awards
Grants & Awards
NRF Master's Research Grant (Principal Investigator)
- Competitive individual research grant from the National Research Foundation of Korea (NRF) under the Ministry of Education's Basic Research Program for Master's Students
- Project: Multimodal-based Video Summarization and Language Model Evaluation Benchmark
Technical Skills
Technical Skills
Programming & Frameworks: Python, PyTorch, HuggingFace, C/C++, Bash, LATEX
Architectures: LLMs, Multimodal LLMs, Vision Transformers, Diffusion Models, GANs
Training & Optimization: Contrastive learning, instruction fine-tuning, preference optimization (RLHF/DPO), cross-modal alignment, distributed training (DDP)
Retrieval & Data: RAG, FAISS, SQL
Infrastructure & Tools: Linux, Slurm, Docker, Git/GitHub, Weights & Biases
Undergraduate Projects
Undergraduate Projects
Exploratory projects from my undergraduate studies, including independent experiments and toy implementations.
Sketch-Text based Image Retrieval for Fashion Products Jul 2024- Aug 2024 Github • Created a sketch-image paired dataset from fashion product images using edge maps and various post-processing techniques • Applied contrastive learning for sketch-text-image representation with CLIP
Outpainting with Edge Generation Mar 2024– Jun 2024 Github • Proposed a model architecture to enhance outpainting performance using Attention GAN and Repaint DDPM
DPO-based Korean Language Detoxification with Context-Appropriate EmojisMar 2024– Jun 2024 • Generated a 38k triplet(toxic-cleaned-cleaned with emojis) dataset in Korean using LLM • Utilized Direct Preference Optimization approach for language detoxiification
Improved Anticipate Accidents in Dashcam Videos Feb 2024– Apr 2024 Github • Enhanced the performance of the original paper by optimizing LSTM with a visual attention mechanism• Modified the model architecture and loss function to improve training stability
Fine-Tuning CLIP with Few-Shot Learning: Impact of Momentum on Performance Oct 2023– Dec 2023 Github • Improve the performance of few-shot learning in CLIP and deeply explored the impact of momentum in model performance
Page updated
Google Sites
Report abuse