Last updated: November, 2025
CV | Google Scholar | LinkedIn | Github
Embedded Files
Jihyun Lee
Hello! I am a Ph.D. candidate at POSTECH (Pohang University of Science and Technology), advised by three wonderful professors Prof. Gary Geunbae Lee, Prof. Yunsu Kim, and Prof. Hyounghun Kim in NLPlab@POSTECH.
My research aims to develop goal-oriented and socially aligned conversational intelligence. I am particularly interested in dialogue generation and evaluation, LLM alignment and personalization, and reinforcement learning–based simulation environments for improving real-world conversational agents, especially in counseling and mental health support contexts.
Contact: jihyunlee [at] postech.ac.kr
📰 News
2025-10-27: Our keyword-based ASR error augmentation "Speak & Spell" has been accepted to AACL-IJCNLP 2025 Main. ✨
2025-08-21: Our LLM simulation based counselor model "Panic to Calm" has been accepted to EMNLP 2025 Main.✨
2025-08-21: Our multimodal counseling dialogue "Mirror" has been accepted to EMNLP 2025 Main. ✨
2025-07-16: Our preference-based dialogue clustering has been accepted to the DSTC12@SIGDIAL 2025 (paper).
2025-01-23: Our multimodal personalized TOD dataset "PicPersona-TOD" has been accepted to NAACL 2025 Main.✨
2025-01-23: Our dialogue-context TTS model has been accepted to NAACL 2025 Findings (paper).
📌 Research Highlights
PanicToCalm: A Proactive Counseling Agent for Panic Attacks (EMNLP 2025, link)
Jihyun Lee, Yejin Min, San Kim, Yejin Jeon, SungJun Yang, Hyounghun Kim, Gary Lee
Can an AI counselor truly help someone in the midst of a panic attack?
This question motivated our work on PanicToCalm, which introduces PACE, a dataset of high-distress episodes constructed from first-person narratives, and PACER, a counseling model designed to deliver real-time Psychological First Aid (PFA). Trained within a simulation environment using the DPO algorithm, PACER demonstrates proactive and emotionally grounded intervention during real psychological emergencies. Together, this work provides a blueprint for developing LLM-based counseling agents capable of responsive and human-aligned crisis support.
MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance (EMNLP 2025, link)
Subin Kim*, Hoonrae Kim*, Jihyun Lee*, Yejin Jeon*, Gary Lee (* Equal contribution)
Can facial expressions influence how counselors understand and respond to client resistance — and if so, how can we address the lack of multimodal data for such complex interactions?
To address this challenge, we present MIRROR, a synthesized multimodal counseling dataset that combines facial expressions, textual dialogue, and resistance-type annotations. Using this dataset, we train a vision–language counseling model that learns to interpret client resistance and adjust counselor responses accordingly.
Experimental results show that incorporating facial cues significantly enhances the model’s ability to detect resistance and generate adaptive, empathetic interventions.
PicPersona-TOD: A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona (NAACL 2025, link)
Jihyun Lee, Yejin Jeon, Seungyeon Seo, Gary Geunbae Lee
How can we make task-oriented dialogue agents respond with personality and style, rather than generic, flat utterances?
To answer this, we introduce PicPersona-TOD, a 🖼️multimodal dataset that enriches user personas with images and contextual attributes such as age or emotional state. Leveraging this, we train Pictor, a vision-language model that generates personalized responses aligned with visual persona cues.
Human evaluations show that incorporating image-based persona information significantly improves engagement and response appropriateness across previously unseen domains.
📚 Publications
Speak & Spell: LLM-Driven Controllable Phonetic Error Augmentation for Robust Dialogue State Tracking
Jihyun Lee, Solee Im, Wonjun Lee, Gary Geunbae Lee
IJCNLP-AACL Main, 2025
# DST # ASR-error # Robustness
linkPanicToCalm: A Proactive Counseling Agent for Panic Attacks
Jihyun Lee, Yejin Min, San Kim, Yejin Jeon, SungJun Yang, Hyounghun Kim, Gary Lee
Empirical Methods in Natural Language Processing (EMNLP), Main, 2025
# Dialogue Model # Counseling # Dialogue Synthesis # Simulation-DPO
linkMIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance
Subin Kim*, Hoonrae Kim*, Jihyun Lee*, Yejin Jeon*, Gary Lee (* Equal contribution)
Empirical Methods in Natural Language Processing (EMNLP), Main, 2025
# Dialogue Model # Counseling # Dialogue Synthesis # Multi-Modal
linkProgressive Facial Granularity Aggregation with Bilateral Attribute-based Enhancement for Face-to-Speech Synthesis
Yejin Jeon, Youngjae Kim, Jihyun Lee, Hyounghun Kim, Gary Lee
Empirical Methods in Natural Language Processing (EMNLP), Findings, 2025
# TTS #Face-to-Speech
linkThe Limits of Post-hoc Preference Adaptation: A Case Study on DSTC12 Clustering
Jihyun Lee, Gary Lee
The Twelfth Dialog System Technology Challenge (DSTC12) Workshop @ SIGDIAL 2025
# Dialogue Theme Clustering
linkPicPersona-TOD: A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona
Jihyun Lee, Yejin Jeon, Seungyeon Seo, Gary Geunbae Lee
Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), Main, 2025
# Task-oriented Dialogue # TOD Style-transfer # Dataset Synthesis # Multi-Modal
linkPrompt-Guided Selective Masking Loss for Context-Aware Emotive Text-to-Speech
Yejin Jeon, Youngjae Kim, Jihyun Lee, Gary Geunbae Lee
Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), Findings, 2025
# TTS # TTS with Dialogue
linkExploring the Viability of Synthetic Audio Data for Audio-Based Dialogue State Tracking
Jihyun Lee*, Yejin Jeon*, Wonjun Lee, Yunsu Kim and Gary Geunbae Lee
IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2023
# DST # ASR # Synthesized Audio
linkTracking Must Go On : Dialogue State Tracking with Verified Self-Training
Jihyun Lee, Chaebin Lee, Yunsu Kim and Gary Geunbae Lee
Interspeech, 2023
# DST
linkDORIC : Domain Robust Fine-Tuning for Open Intent Clustering through Dependency Parsing
Jihyun Lee, Seungyeon Seo, Yunsu Kim, Gary Geunbae Lee
The Eleventh Dialog System Technology Challenge (DSTC11) Workshop at SIGDIAL & INLG 2023
# Dialogue Theme Detection
linkExploring Back Translation with Typo Noise for Enhanced Inquiry Understanding in Task-Oriented Dialogue
Jihyun Lee, Junseok Kim, Gary Geunbae Lee
The Eleventh Dialog System Technology Challenge (DSTC11) Workshop at SIGDIAL & INLG 2023
# Task-oriented Dialogue # TOD # Augmentation
link
SF-DST: Few-Shot Self-Feeding Reading Comprehension Dialogue State Tracking with Auxiliary Task
Jihyun Lee, Gary Geunbae Lee
Interspeech, pp. 1233-1237, 2022
# DST
link
🏆Awards
Global Innovation Award, ICT Challenge 2025 (Minister of Science and ICT Award)
Jihyun Lee, Wonjun Lee, Sungjun Yang – Multi-modal, Multi-session Counseling System
Ministry of Science and ICT, Republic of Korea (과학기술정보통신부 장관상), 2025
🖼️ Photos
Page updated
Google Sites
Report abuse