Research Interests

Multimodal Learning

Self-supervised Learning

Social Artificial Intelligence

Experience

Sungkyunkwan University (SKKU), Seoul, South Korea (Sep 2024 - Present) 

University of Illinois Urbana-Champaign (UIUC), IL, United States (May 2023 - Aug 2024) 

Georgia Institute of Technology (Georgia Tech), GA, United States (Oct 2023 - Aug 2024) 

Education

Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea (Feb 2023) 

Yonsei University, Seoul, South Korea (Feb 2017) 

Publications

1. MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization

2. SocialGesture: Delving into Multi-person Gesture Understanding

3. Object-aware Sound Source Localization via Audio-Visual Scene Understanding

4. Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders

5. Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation

6. Question-Aware Gaussian Experts for Audio-Visual Question Answering

7. Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection

8. Towards Social AI: A Survey on Understanding Social Interactions

9. Leveraging Object Priors for Point Tracking

10. Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations

11. Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge

12. Text-Guided Distillation Learning to Diversify Video Embeddings for Text-Video Retrieval

13. Analyzing Visible Articulatory Movements in Speech Production for Speech-Driven 3D Facial Animation

14. Audio-Visual Mismatch-Aware Video Retrieval via Association and Adjustment

15. Weakly Paired Associative Learning for Sound and Image Representations via Bimodal Associative Memory

16. Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning

17. CUA Loss: Class Uncertainty-Aware Gradient Modulation for Robust Object Detection

18. Assessing Individual VR Sickness through Deep Feature Fusion of VR Video and Physiological Response

19. Towards a Better Understanding of VR Sickness: Physical Symptom Prediction for VR Contents

20. Visual Comfort Aware-Reinforcement Learning for Depth Adjustment of Stereoscopic 3D Images

21. SACA Net: Cybersickness Assessment of Individual Viewers for VR Content via Graph-based Symptom Relation Embedding

22. Structure Boundary Preserving Segmentation for Medical Image With Ambiguous Boundary

23. Video Frame Interpolation via Exceptional Motion-aware Synthesis

24. BMAN: Bidirectional Multi-scale Aggregation Networks for Abnormal Event Detection

25. Robust Video Frame Interpolation with Exceptional Motion Map

26. Estimating VR Sickness Caused by Camera Shake in VR Videography

27. Physiological Fusion Net: Quantifying Individual VR Sickness with Content Stimulus and Physiological Response

28. Deep Objective Assessment Model Based on Spatio-temporal Perception of 360-Degree Video for VR Sickness Prediction

29. VRSA Net: VR Sickness Assessment considering Exceptional Motion for 360-degree VR Video

30. On-the-Fly Facial Expression Prediction using LSTM Encoded Appearance-Suppressed Dynamics

31. STAN: Spatio-temporal Adversarial Networks for Abnormal Event Detection

Patents

1. Method for Video Frame Interpolation Robust to Exceptional Motion and the Apparatus Thereof

Korea Patent 2244187 / PCT Patent App. 003461 

2. Method for VR Sickness Assessment Considering Neural Mismatch Model and the Apparatus Thereof

Korea Patent 2284266 / US Patent 11699072 

3. Apparatus and Method for Virtual Reality Sickness Reduction Based on Virtual Reality Sickness Assessment

Korea Patent 2291257 / US Patent 11252371  

4. Video Sequences Generating System Using Generative Adversarial Networks and the Method Thereof

Awards & Honors

BMVC Outstanding Reviewer (2024) 

CVPR Oral Presentation (Top 0.8% of Submissions) (2024) — 1st-Author 

1st Place Winner of Ad-hoc Video Search Competition (2022) — Team Lead

CVPR Oral Presentation (Top 4% of Submissions) (2021) — 1st-Author

Samsung HumanTech Paper Award, Honor Prize ($2,000) (2021) — 1st-Author

Outstanding Project Selection (2020) — Graduate Project Lead

Outstanding TA Award (2 Times) (2019, 2020) — Head TA

ICIP Best Paper Finalist (Top 1% of Submissions) (2019) — 1st-Author

ICIP Top 10% Paper Selection (2019) — 1st-Author

National Government Fellowship (2017 - 2023) 

Professional Activities

Organizer

Reviewer

Invited Talks

Teaching