Doyoung Gwak

I am currently working on optimizing models by applying quantization techniques at Google in South Korea. This area is really interesting to me because it helps reduce the costs associated with machine learning and improves user experiences. 

I've also been deeply interested in on-device machine learning (ODML) for a long time. This passion has been a significant influence throughout my career. I started out in iOS app development and later expanded my skills to include server-side machine learning engineering.

Work Experience

2022.09~Present Google LLC

Optimize Server-side models via quantization techniques (Python, C++, and MLIR)

• Develop the Quantization Debugging feature for Server-Side models [commit]

Optimize ODML(On-device ML) models via quantization techniques (Python, C++, and MLIR)

• Reinforce and maintain PTQ, QAT conversion pipeline for TensorFlow Lite

• Improve the performance of TensorFlow Lite converter (x2 reduction of large model conversion)

• Maintain the TensorFlow Lite converter, working with both MLIR manipulation level and FlatBuffer manipulation level [commits]

Fundamental effort (MLIR)

• Support MLIR for netron.app model visualization (PR)

2019.09~2022.08 NAVER corp. 

Develop and apply ML in real service in NAVER, like LINE CONOMI in Japan

Restaurant Atmosphere Image Classification (Vision, ML, Image Classification, Semi-supervised Learning, Pytorch)

• Make a POC classifier with a sample dataset with teachable machine

• Boost 4x times speed of building image classification annotation by pre-annotating with PoC classifier

• Make a baseline code for image classification, train the own dataset, and do hyperparameter tuning

Image Rotation Detector in Server (Vision, ML, Image Classification, TensorFlow)

• Collect and refine the dataset by using ML (using teachable machine's prototype model)

• Train with models provided by Keras's applications and serve the trained model with Flask and gunicorn

• Implement a Kafka consumer and ad-hoc batch to use the API

Receipt Image Edge Detector on iOS (Vision, ML, On-device ML, Pose Estimation, TensorFlow, TensorFlowLite)

• Refine the receipt's 4 key points annotated dataset with ML (annotate and infer iteratively) with mobile annotation tool, which I made

• Train the model with tf2-mobile-pose-estimation's models

• Convert the TF2 model to the TFLite model and make a prototype application using the model

  • Show the heat-map output of the model on the mobile device

  • Recognize a receipt in a camera, whether it could be automatically captured or not (pre-processing of OCR request)

[intern] Food Image Detector Album  on iOS (Vision, ML, On-device ML, Binary Image Classification, Create ML, Core ML)

• Train a food or not classifier with an open dataset using Create ML

• By using the model, implement a photo album that only shows food images

• Implement user interactive inference batch in the background without UI blocking

Education

• 2012−2020: BS, Computer Science Engineering, Pusan National University, Pusan, Republic of Korea. GPA 3.48

Personal Projects

2021 − "TFLiteSwift-Vision" Repository (iOS, On-device ML, TFLiteSwift, Modularization, CocoaPods) [Github Link]

Provide a vision domain specific-library that implements pre/post-processing of vision task

• The reason why I am making TFLiteSwift-Vision

• Implement various preprocessing functions for vision tasks

• Make use-cases in various TFLiteSwift demo projects

2021 − "Golf Swing Clipper" App (iOS, On-device ML, Core ML, Vision framework, Pose Estimation) [App Store Link]

Develop a golf swing clipping app using MLKit's 3D Pose Estimation

• When the user imports a long golf swing video, it inference every frame to the skeleton asynchronously

• In the video player preview, the user can check the skeleton for the current video time

• Implement rule base feature to detect golf swing time range with the sequence of the skeleton that estimates with 3d pose estimation

• Implement interpolation for the sequence of keypoint and make skeleton movement smooth during the user's seeking control

2021 − Contribution to huggingface/transformers (NLP, GPT2, NER) [PR Link]

Make PR for GPT2ForTokenClassification class which configures by GPT2 for upstream and NER for downstream task

2020 − "PoseEstimation-TFLiteSwift" Repository (iOS, On-device ML, TFLiteSwift, Pose Estimation, Modularization) [Github Link]

Provide a sample source code for developers to easily use single and multi-person pose estimation models with Swift and TFLiteSwift

• Implement OpenPose paper's multi-person pose estimation post-processing with Swift by using a bipartite graph, line integral, non-maximum suppression, etc.

• Design extensible code architecture for supporting various shapes of outputs of pose estimation models(OpenPose, PoseNet, Hourglass, and CPM)

• Implement drawing heat map(PCM and PAF) on the screen for the model's output debugging

• Implement 3D pose estimation pre-processing like soft-argmax in Swift

• Implement to draw 3d key points on the screen with SceneKit and render the key points in real-time

2020 − "SemanticSegmentation-CoreML" Repository (iOS, On-device ML, Core ML, Metal, Semantic Segmentation) [Github Link]

Provide a sample source code for developers to easily use the semantic segmentation model with Swift and Core ML framework

• Implement to draw segmentation-map with CPU(CoreGraphics framework) and GPU(Metal framework), and compare each performance

2020 − "Capture& Paste" App (macOS, On-device ML, Core ML, Vision framework, OCR) [App Store Link]

Develop a simple utility macOS app using Text Recognition with Vision framework and open-source libraries

2019 − "Just Point It" App (iOS, Vision, On-device ML, ML Kit, Core ML, Models' Pipeline, Fingertip Pose Estimation, OCR, Post-processing Optimization) [App Store Link]

Design, develop and release the "Just Point It" app using vision-based 3 machine learning models and optimization algorithms

• Implement the pipeline of image data through pose estimation(Core ML), text detection(Vision framework), and text recognition(ML Kit) models

• Use the majority voting algorithm to improve the performance

2019 − "PoseEstimation-CoreML" Repository (iOS, On-device ML, Core ML, Pose Estimation, Pose Matching) [Github Link]

Provide a sample source code for developers to easily use pose estimation model with Swift and Core ML framework

• Implement converting logic from camera frame to pixel buffer for pre-processing and argmax for converting heat-map to joint location for post-processing

• Make pose matching algorithm which scores for comparing between target pose and inferred pose

• Implement to draw a heatmap (or heatmaps) on the screen for model's output debugging

2018 − "awesome-ml-demos-with-ios" Repository [Github Link]

Collect implementation of various pre/post-processing of each vision model on iOS

• My implementations included in:
  DepthPrediction-CoreML, ImageClassification-CoreML, ObjectDetection-CoreML, TextDetection-CoreML, TextRecognition-MLKit, FaceDetection-MLKit, etc.

• Measure the latency of the inference time and post-processing time with 3 ways on mobile device

• Compare the latencies between the iOS devices each model

Other Projects

• 2019 − ReHapp (iOS, Pose Estimation, Pose Matching) [Github Link]

• 2018 − Don't Be Turtle (iOS, Pose Estimation) [Github Link]

• 2016 − Roast Meister (iOS, iPhone&iPad) [App Store Link]

• 2014 − Hangul Clock (iOS, iPhone&iPad) [App Store Link]

• 2014 − Guitar Kit+ (iOS, Cocos2D Engine) [App Store Link]

• 2013 − Easi Calc (iOS) [App Store Link(free)] [App Store Link(Pro)] [App Store Link(Widget)]

Publications & Blog Posts & Presentations

• 2022.11 − Korean translation book of "AI and Machine Learning for On-Device Development", with co-translator Chansung Park [yes24 link(kr)]

• 2021.08 − Find a restaurant with a good night view in Naver service (Vision, Image Classification, Semi-supervised learning, Hierarchical Classification) [Blog Post Link(kr)]

• 2020.11 − Participate in Core ML Panel, Let's Swift 2020 Online Lightning Talk [Video Link(kr)]

• 2020.09 − Annotate 40k images while you don't get bored (with ML), Medium [Blog Post Link(kr)]

• 2020.08 − 2020 What's New in Apple's Machine Learning, presentation in ApplicableML community [Slide Link(kr)]

• 2019.12 − Will my ML model work on the mobile app, MODUCON 2019 Conference [Slide Lin(kr)], [Video Link(kr)]

• 2019.11 − Problems that machine learning can solve in iOS apps, Let's Swift 2019[Slide Link(kr)], [Video Link(kr)]

• 2019.08 − Machine Learning with Apple, special lecture presentation in Dongguk University [Slide Link(kr)]

• 2019.05 − "Just Point It": Machine Learning on iOS with Pose Estimation + OCR using Core ML and ML Kit, Medium [Blog Post Link(en)]

• 2019.03 − "Just Point It", A dictionary that finds you when you point it out, NAVER TechTalk [Slide Link(kr)], [Video Link(kr)]

• 2018.06 − Core ML 맛보기, MotLabs community [Slide Link(kr)]

Community Activity

• 2020~Present − Applicable ML [Community Blog]

  • With the people who are interested in mobile application + ML, develop personal projects and share what we learn 

  • Participate as a community leader

• 2018~2019 − MoT Labs [Community Blog]

  • Participate as an iOS developer

Honor & Awards

• 2019 − 6th NAVER D2 CAMPUS FEST, 1st Place Award

• 2016 − Guitar Kit+ featured on App Store, "Learn to Play"

• 2015 − 2015 Startup Weekend Busan, Best Engineering Award

• 2015 − Guitar Kit+ featured on App Store, "New Year, New You", "Indoor leisure activities" and "Best Music"

• 2014 − Guitar Kit+ featured on App Store, "Weekly Best"