Minsu Kim

Publications

International Journal

1. AKVSR:  Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model

Jeong Hun Yeo, Minsu Kim, Jeongsoo Choi, Dae Hoe Kim, and Yong Man Ro

IEEE Transactions on Multimedia (TMM), 2024 [Paper]

2. Cromm-vsr: Cross-modal memory augmented visual speech recognition

Minsu Kim, Joanna Hong, Se Jin Park, and Yong Man Ro

IEEE Transactions on Multimedia (TMM), 2021 [Paper] [Code]

3. Speech Reconstruction with Reminiscent Sound via Visual Voice Memory 

Joanna Hong, Minsu Kim, Se Jin Park, and Yong Man Ro

IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 2021 [Paper]

International Conference

1. AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation

Jeongsoo Choi*, Se Jin Park*, Minsu Kim*, and Yong Man Ro (* Co-First Authors)

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. (Highlight) [Paper] [Code] [Demo]

2. Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens

Minsu Kim, Jeongsoo Choi, Soumi Maiti, Jeong Hun Yeo, Shinji Watanabe, and Yong Man Ro

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024 [Paper] [Code] [Demo]

3. Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper

Jeong Hun Yeo*, Minsu Kim*, Shinji Watanabe, and Yong Man Ro (* Co-First Authors)

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024 [Paper] [Data]

4. Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models

Jeongsoo Choi, Minsu Kim, Se Jin Park, and Yong Man Ro

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024 [Paper]

5. Exploring Phonetic Context-Aware Lip-Sync for Talking Face Generation

Se Jin Park, Minsu Kim, Jeongsoo Choi, and Yong Man Ro

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024 [Paper]

6. Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge

Minsu Kim*, Jeong Hun Yeo*, Jeongsoo Choi, and Yong Man Ro (* Co-First Authors)

IEEE/CVF International Conference on Computer Vision (ICCV), 2023 [Paper]

7. Intelligible Lip-to-speech Synthesis with Speech Units 

Jeongsoo Choi, Minsu Kim, and Yong Man Ro

24th INTERSPEECH Conference, 2023 [Paper] [Code] [Demo]

8. Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring 

Joanna Hong*, Minsu Kim*, Jeongsoo Choi, and Yong Man Ro (* Co-First Authors)

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023 [Paper] [Code] [Data]

9. Lip-to-speech Synthesis in the Wild with Multi-task Learning

Minsu Kim*, Joanna Hong*, and Yong Man Ro (* Co-First Authors)

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023 [Paper] [Code]

10. Multi-Temporal Lip-Audio Memory for Visual Speech Recognition

Jeong Hun Yeo, Minsu Kim, and Yong Man Ro

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023 [Paper]

11. Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video

Minsu Kim, Chae Won Kim, and Yong Man Ro

AAAI Conference on Artificial Intelligence (AAAI), 2023 [Paper]

12. Speaker-adaptive Lip Reading with User-dependent Padding

Minsu Kim, Hyunjun Kim, and Yong Man Ro

European conference on computer vision (ECCV), 2022 [Paper] [Data] [Code]

13. VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection

Joanna Hong, Minsu Kim, and Yong Man Ro

European conference on computer vision (ECCV), 2022 [Paper]

14. Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition

Joanna Hong*, Minsu Kim*, Daehun Yoo, and Yong Man Ro (* Co-First Authors)

23rd INTERSPEECH Conference, 2022 [Paper] [Code] 

15. Distinguishing Homophenes using Multi-head Visual-audio Memory for Lip Reading

Minsu Kim, Jeong Hun Yeo, and Yong Man Ro

AAAI Conference on Artificial Intelligence (AAAI), 2022 [Paper] [Code]

16. SyncTalkFace: Talking Face Generation with precise Lip-syncing via Audio-Lip Memory

Se Jin Park, Minsu Kim, Joanna Hong, Jeongsoo Choi, and Yong Man Ro

AAAI Conference on Artificial Intelligence (AAAI), 2022 [Paper]

17. Lip to Speech Synthesis with Visual Context Attentional GAN

Minsu Kim, Joanna Hong, and Yong Man Ro

Advances in Neural Information Processing Systems (NeurIPS), 2021 [Paper] [Code]

18. Multi-modality associative bridging through memory: Speech sound recollected from face video

Minsu Kim*, Joanna Hong*, Se Jin Park, and Yong Man Ro (* Co-First Authors)

IEEE/CVF International Conference on Computer Vision (ICCV), 2021 [Paper] [Code]

19. Interpretation of Lesional Detection via Counterfactual Generation

Junho Kim, Minsu Kim, and Yong Man Ro

IEEE International Conference on Image Processing (ICIP), 2021 [Paper]

20. Unsupervised Disentangling of Viewpoint and Residues Variations by Substituting Representations for Robust Face Recognition

Minsu Kim, Joanna Hong, Junho Kim, Hong Joo Lee, and Yong Man Ro

International Conference on Pattern Recognition (ICPR), 2020 [Paper]

21. Robust video facial authentication with unsupervised mode disentanglement

Minsu Kim, Hong Joo Lee, Sangmin Lee, and Yong Man Ro

IEEE International Conference on Image Processing (ICIP), 2020 [Paper]

22. Learning Style Correlation for Elaborate Few-Shot Classification

Junho Kim, Minsu Kim, Jung Uk Kim, Hong Joo Lee, Sangmin Lee, Joanna Hong, and Yong Man Ro

IEEE International Conference on Image Processing (ICIP), 2020 [Paper]