Sparse autoencoders reveal selective remapping of visual concepts during adaptation 

• Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider.

• Here we develop a new Sparse Autoencoder (SAE) for the CLIP vision transformer, named PatchSAE, to extract interpretable concepts at granular levels (e.g., shape, color, or semantics of an object) and their patch-wise spatial attributions.
  [PDF] [code] [poster & video] [website & 🤗 demo] 

• ICLR 2025 Accepted. Presented in Singapore 🇸🇬

Towards Calibrated Robust Fine-Tuning of Vision-Language Models

• Changdae Oh*, Hyesu Lim*, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jaegul Choo, Alexander G Hauptmann, Zhi-Qi Cheng, Kyungwoo Song. (*: equal contributions) 

• This work proposes a robust fine-tuning method that improves both OOD accuracy and calibration error in Vision Language Models (VLMs) via fine-tuning with a constrained multimodal contrastive loss that minimizes a shared upper bound of OOD generalization and calibration error. [PDF] [Code]

• NeurIPS 2024 Accepted. Presented in Vancouver, Canada 🇨🇦

• NeurIPSW 2023 (DistShift) Accepted. Presented in New Orleans, USA 🇺🇲

TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptation

• Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. 

• We propose a test-time batch normalization method, which interpolates source and current batch statistics considering each layer's domain-shift sensitivity level, showing robust performance over various realistic evaluation scenarios. [PDF] [website]

• ICLR 2023 Accepted. Presented in Kigali, Rwanda 🇷🇼

AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain 

• Jimin Hong*, Taehee Kim*, Hyesu Lim*, and Jaegul Choo. (*: equal contributions)

• We propose to consider a vocabulary of a pre-trained language model as an optimizable parameter, allowing us to update the vocabulary by expanding it with domain-specific vocabulary based on tokenization statistics. [PDF] [Code] [Video&Slide]

• EMNLP 2021 Accepted. Presented in Punta Cana, Dominican Republic 🇩🇴