Jongju Shin

[Information]

Multimodal Data,

Kakao Corp,

166, Pangyoyeok-ro, Bundang-gu, Seongnam-si, Gyeonggi-do, 13529, Korea

E-mail : isaac.j.shin@gmail.com,

jjshin@postech.ac.kr,

isaac.newton@kakaocorp.com

[Educational Background]

1999.03~2007.02 B.S., Dept. of Information & Computer Engineering, Ajou University, Suwon, South Korea.

2007.02~2015.02, Ph. D., Dept. of Computer Science & Engineering, POSTECH, Pohang, South Korea.

2011.09~2012.08 Visiting Student, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, US

2015.01~2015.05, Post-Doc., Dept. of Computer Science & Engineering, POSTECH, Pohang, South Korea.

[Work Experience]

2002.04~2003.04 EBI solution (closed)

2003.05~2004.05 Goldenpond (closed)

2004.05~2006.02 Unidocs

2015.06~2019.12, Research Engineer, Kakao Corp., South Korea.

2019.12~2021.07, Research Engineer, Kakao Enterprise, South Korea.

2021.08~2023.07, Vision Intelligence Team Lead, Kakao Enterprise, South Korea.

2023.08~2024.05, Vision Team Lead, KakaoBrain, South Korea.

2024.06~current, Multimodal data, Kakao Corp., South Korea.

[Research Interest]

Face related problems - detection, alignment(including tracking), recognition, attribute(expression, gender, age, etc) estimation.

Face Recognition (Kakao Enterprise, Kakao)

ranked 1st place in FRVT 1:N investigation - kiosk task by July 2022. (report)
ranked 1st place and 2nd place in FRVT 1:1 kiosk task and border task by June 2022.

Interview

https://www.youtube.com/watch?v=3ymbVf8d6vs
https://www.mk.co.kr/news/it/10438680

ranked 2nd place in WebFace260M track of ICCV2021 Masked Face Recognition Challenge (MFR). (link)
ranked 3rd place in FRVT 1:1 wild task @ FMR=0.0001 by Mar. 25. 2020 report. (report, blog)
Our face recognition is used in daum image search to cluster celebrities' images. (more than 50000 peoples)

Facial Feature Tracking in Mobile Device (Kakao, closed service)

Our goal is to develop facial feature tracking algorithm which runs on mobile devices.
Our algorithm is used in kakaotalk cheez.

Daum roadview blurring

To blur pedestrians and car plates, we developed face detection, and car plat detection algorithms. (link)

Celeb Image Ranking Model

Rank images according to the qualities such as the number of faces.

Facial Feature Detection and Tracking (POSTECH)

Hybrid approach to combine local and global methods.

Facial Feature Detection (CMU)

All pose face alignment robust to occlusion (link)

Based on multiple shape models and RANSAC framework, we can align face robust to occlusion.

This work is done with Takeo Kanade and Myung-Cheol Roh in CMU.

Facial Feature Tracking (POSTECH)

Based on Active Appearance Models, we add global and local template matching, template update, and motion estimation.

We compared our algorithm with others. The result on the right bottom is ours.

In detail, We show the result with big head with mouth motion.

Facial Feature Tracking and Head Pose Estimation (POSTECH)

We combined cylindrical head model and facial feature tracking(based on Inverse Compositional image alignment algorithm with face shape model). The technique of facial feature tracking here is used for lip reading system in my journal.

[Publications]

International Journal

Jongju Shin, and Daijin Kim, "Robust Face Alignment and Tracking by Combining Local Search and Global Fitting ", Image and Vision Computing, 2016

Jongju Shin, and Daijin Kim, "Hybrid approach for facial feature detection and tracking under occlusion", IEEE Signal Processing Letters, Vol. 21, Issue. 12, pp.1486-1490, 2014

Jongju Shin, Jin Lee, and Daijin Kim, "Real-time lip reading system for isolated Korean word recognition", Pattern Recognition, Vol. 44, Issue. 3, pp. 559-571, Mar. 2011.

Technical Report

Minseong Kim, Jongju Shin, Myung-Cheol Roh, Hyun-Chul Choi, "Uncorrelated Feature Encoding for Faster Image Style Transfer", arxiv, Jul 2018 (link)
Vishnu Naresh Boddeti, Myung-Choel Roh, Jongju Shin, Takaharu Oguri, and Takeo Kanade, “Face Alignment Robust to Pose, Expressions and Occlusions”, arxiv, Jul 2017 (link)
윤재삼, 양정석, 신종주, "[카카오 AI 리포트] NIPS에서 발견한 AI 트렌드", Jan 2018 (link)
Myung-cheol Roh, Pyoung-gang Lim, Jongju Shin, "Gallery Sampling for Robust and Fast Face Identification", arxiv, May 2023 (link)

International Conference

Yonghyun Kim, Wonpyo Park, and Jongju Shin , "BroadFace: Looking at Tens of Thousands of People at Once for Face Recognition", In Proceeding of the European Conference On Computer Vision (ECCV) 2020, (link) (arxiv link)
Yonghyun Kim, Wonpyo Park, Myung-Cheol Roh, and Jongju Shin, "GroupFace: Learning Latent Groups and Constructing Group-based Representations for Face Recognition", In Proceeding of the IEEE Computer Vision and Pattern Recognition (CVPR), 2020 (cvf link)(arxiv link)
Jongmin Yoon, Jongju Shin, and Daijin Kim , "Multi-pose face detection for silver robots" , In Proceeding of the IEEE Ubiquitous Robots and Ambient Intelligence (URAI), pp.755-579, 2013.
Sangseok Yun, Jongju Shin, Daijin Kim, Chang Gu Kim, Munsang Kim, and Mun-Taek Choi, "Engkey: Tele-education Robot” Social Robotics", In Proceeding of the Third international conference on Social Robotics, pp.142-152, 2011
Jongju Shin, Bongjin Jun, and Daijin Kim, "Robust two-stage lip tracker", In Proceeding of the IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), pp. 265-270, 2009.
Jiman Kim, Jongju Shin and Daijin Kim, "FH-adaptive motion compensation for moving object detection in an active camera", In Proceeding of the IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), pp. 532-537, 2009.
Kyung-geune Oh, Chan-yul Jung, Won-hyo Yu, Jong-ju Shin, Daijin Kim and Seung-jong Kim, "Real Time Imitation of Facial Expression with a Mascot Type Face Robot", In Proceeding of the 14th International Conference on Advanced Robotics (ICAR2009), Germany, Jun, 2009.
Jongju Shin, Daijin Kim, "Enhanced Resolution-Aware Fitting Algorithm using Interpolation Operator", In Proceeding of the 19th International Conference on Pattern Recognition (ICPR2008), US, Dec, 2008.
Sungsoo Park, Jongju Shin, and Daijin Kim, "Facial expression analysis with facial expression deformation", In Proceeding of the 19th International Conference on Pattern Recognition (ICPR2008), US, Dec, 2008.
Hyung-Soo Lee, Sungsoo Park, Bong-Nam Kang, Jongju Shin, Ju-Young Lee, Hongmo Je, Bongjin Jun, and Daijin Kim, "The POSTECH Face Database (PF07) and Performance Evaluation", In Proceeding of the 8th IEEE International conference on Automatic Face and Gesture recognition, Amsterdam, the netherlands, 2008.
Jongju Shin, Daijin Kim, "Facial Expression Recognition using Feature Tracking", In Proceeding of the 8th POSTECH-KYUTECH Joing Workshop, JAPAN, Aug, 2008.

Domestic Journal

Hyunsung Park, Jongju Shin, Jieun Kim, Daijin Kim , "휴대 단말기를 위한 고속 Active Appearance Model" , 정보과학회 논문지 : 컴퓨팅의 실제 및 레터 제 20권 제 6호, 2014.

Domestic Conference

Jongju Shin, "휴대 단말기에서의 고속 얼굴 특징점 추적", IPIU, 2017
Jongju Shin, Daijin Kim, "MCT 및 Adaboost를 이용한 다양한 인종의 성별인식", 정보과학회 추계학술대회, 2013
Hyunsung Park, Jongju Shin, Jieun Kim, Daijin Kim, "휴대 단말기를 위한 고속 Active Apperance Model", 정보과학회 추계학술대회, 2013
Yoonjung Hong, Jongju Shin, Bongnam Kang, Daijin Kim, "피라미드 가버 필터와 계층적 분류기의 나이 추정 방법", 정보과학회 추계학술대회, 2013
Hyunsung Park, Jongju Shin, Daijin Kim, "Occlusion robust active appearance model with validating meshes", IPIU, 2013
Jongju Shin, Daijin Kim, "Facial feature tracking robust to face movement", IPIU, 2013
Inho Choi, Jongju Shin, Daijin Kim, "자연스런 영상통화를 위한 정면 얼굴영상에서의 시선보정", IPIU, 2012
Jongju Shin, Hongil Kim, Soyeon Kang, Sungdae Cho, and Daijin Kim, "Talking face", IPIU, 2011
Jongju Shin, Seungjun Yang, Daijin Kim, and Hankyu Lee, "Eye-contact system based on frontal view synthesis using facial feature extraction", IPIU, 2010
Jongju Shin, Daijin Kim, "휴대 단말기에서의 표정 인식 시스템", IPIU, 2009
Jongju Shin, Daijin Kim, "향상된 Resolution-Aware Fitting을 사용한 저해상도 영상 분석", 제 35회 정보과학회 추계학술대회(10.24~25. 중앙대학교), 2008

[Projects]

- 2014, HYUNDAI Motors, 운전자 시선추적을 위한 얼굴 특징점 검출 기술 개발
- - 차량 환경에서 얼굴 특징점 검출 기술 개발
- 2014, SAMSUNG, 얼굴 특징점 추출 및 변환 IP 개발
- - 영상의 얼굴 특징점 검출 기술 및 Beautifier 기술 개발
- 2013, SAMSUNG, 얼굴 포즈 및 얼굴 관련 신기능 IP 개발
- - 성별 인식 기술 개발
- 2012. KT, Median Intelligence를 위한 객체인식 기술 개발
- - 성별 인식 기술 개발
- 2012. SAMSUNG DMC, 모바일용 특징점 검출 및 추적 기술 개발
- 2012. LG, Development of Gaze Correction for Video Call in Mobile Device
- 2011. SAMSUNG DMC, 휴대 단말용 실시간 시선보정 기술 개발
- 2011. SAMSUNG, 디지털 카메라용 표정 인식 기술 개발
- 2010. ETRI, 화자 기반 아이컨택 영상 가공 기반기술 연구
- 2010. SAMSUNG DMC, 휴대단말에서 얼굴 특징점 추출 기술 개발 및 이를 이용한 Digital Face Enhancer개발
- 2010. NGV, 차량용 표정 인식 기술 개발
- 2010-2013. 노약자를 위한 감성교감 및 이동보조 서비스 로봇 개발
- 2008-2009. LG, 모바일 기기용 웃음 검출기 개발
- 2008. SAMSUNG TN , 휴대 단말용 얼굴 검출, 눈 검출 및 표정 인식 기술 개발
- 2008. SAMSUNG DM , 지능형 포토 관리 시스템 기술 개발
- 2009-2013. KIST, 2D/3D 영상 데이터 기반 휴먼 검출, 얼굴 및 감정 인식 기술 개발
  - 2009 Grand Challenge 요소기술 우수상 - 엘리베이터 버튼 인식 기술 개발
- 2008-2011. 한국과학재단, 노약자의 삶의 질 개선을 위한 휴먼 센싱 기술개발
- 2008-2009. NGV, 음성 인식과 LIP READING 기술을 이용한 네비게이션 제어
- 2008-2009. SAMSUNG, 디지털카메라에서의 얼굴 검출/추적/인식, 표정 /SMILE/BLINK 검출 기술 개발
- 2007-2009. KIST, 얼굴 기반 신원 확인 및 의도 인식 기술
- 2007. ETRI, 원거리 저해상도 영상에서의 얼굴 특징점 추출 기술 개발

[Patents]

- "그룹 기반 얼굴 인식 방법 및 장치",

김용현, 박원표, 노명철, 신종주, 출원번호 10-2020-0059683

- "얼굴 특징점 추출 방법 및 이를 수행하는 장치",

김대진, 신종주, 출원번호 10-2014-0054087, 등록번호 101626050000

- "평균 얼굴 패치 및 메시 기반 나이 추정 시스템 및 방법",

전승선, 홍윤정, 김대진, 신종주, 출원번호 10-2014-0110960

- "립리딩과 음성 인식 통합 멀티모달 인터페이스 시스템",

김대진,이진,신종주,이진석,김대희, 한국, 등록번호 10-1092820 (20111205)

- "정면 얼굴 영상 생성 장치 및 방법",

김대진,신종주,김정태,양승준,임성용,이한규,한국,출원번호 10-2010-0126120 (2010.12.10)

- "사용자 맞춤형 표정 인식 방법 및 장치",

김대진,신종주,천영재,한국,등록번호 10-0988326 (20101011)

- "아이 컨택 시스템을 위한 얼굴 특징점 추출 기반 정면 얼굴 합성",

김대진,신종주,김정태,양승준,임성용,이한규,한국,출원번호 10-2010-0007031 (2010.01.26)

- "COMBINED LIP READING AND VOICE RECOGNITION MULTIMODAL INTERFACE SYSTEM",

Dae Hee Kim, Dai-Jin Kim, Jin Lee, Jong-Ju Shin, Jin-Seok Lee, US, 20110071830 (2011-03-24)

- "Method for correcting user's gaze direction in image, machine-readable storage medium and communication terminal"

Byung-Jun Kim, Dai-Jin Kim, Jong-Ju Shin, In-Ho Choi, Tae-Hwa Hong, US, 20140016871 (2013-02--01)

Page updated

Google Sites

Report abuse