Doyoung Gwak

I am currently working at Google in the Bay Area as part of the TPU Performance Team, developing software that scales quantization techniques. This focus is particularly engaging because it lowers machine learning costs while improving user experiences.

My long-standing interest in on-device machine learning (ODML) has influenced my entire career. It guided my early years as an iOS app developer and later led me to expand my expertise into server-side machine learning engineering.

Work Experience

2022.09~Present Google LLC

Optimize Server-side models via quantization techniques (Python, C++, and MLIR)

• Develop the Quantization Debugging feature for Server-Side models [commit]

Optimize ODML(On-device ML) models via quantization techniques (Python, C++, and MLIR)

• Reinforce and maintain PTQ, QAT conversion pipeline for TensorFlow Lite

• Improve the performance of TensorFlow Lite converter (x2 reduction of large model conversion)

• Maintain the TensorFlow Lite converter, working with both MLIR manipulation level and FlatBuffer manipulation level [commits]

Fundamental effort (MLIR)

• Support MLIR for netron.app model visualization (PR)

2019.09~2022.08 NAVER corp. 

Develop and apply ML in real service in NAVER, like LINE CONOMI in Japan

Restaurant Atmosphere Image Classification (Vision, ML, Image Classification, Semi-supervised Learning, Pytorch)

• Make a POC classifier with a sample dataset with teachable machine

• Boost 4x times speed of building image classification annotation by pre-annotating with PoC classifier

• Make a baseline code for image classification, train the own dataset, and do hyperparameter tuning

Image Rotation Detector in Server (Vision, ML, Image Classification, TensorFlow)

• Collect and refine the dataset by using ML (using teachable machine's prototype model)

• Train with models provided by Keras's applications and serve the trained model with Flask and gunicorn

• Implement a Kafka consumer and ad-hoc batch to use the API

Receipt Image Edge Detector on iOS (Vision, ML, On-device ML, Pose Estimation, TensorFlow, TensorFlowLite)

• Refine the receipt's 4 key points annotated dataset with ML (annotate and infer iteratively) with mobile annotation tool, which I made

• Train the model with tf2-mobile-pose-estimation's models

• Convert the TF2 model to the TFLite model and make a prototype application using the model

  • Show the heat-map output of the model on the mobile device

  • Recognize a receipt in a camera, whether it could be automatically captured or not (pre-processing of OCR request)

[intern] Food Image Detector Album  on iOS (Vision, ML, On-device ML, Binary Image Classification, Create ML, Core ML)

• Train a food or not classifier with an open dataset using Create ML

• By using the model, implement a photo album that only shows food images

• Implement user interactive inference batch in the background without UI blocking

Education

• 2012−2020: BS, Computer Science Engineering, Pusan National University, Pusan, Republic of Korea. GPA 3.48

Personal Projects

2021 − "TFLiteSwift-Vision" Repository (iOS, On-device ML, TFLiteSwift, Modularization, CocoaPods) [Github Link]

Provide a vision domain specific-library that implements pre/post-processing of vision task

• The reason why I am making TFLiteSwift-Vision

• Implement various preprocessing functions for vision tasks

• Make use-cases in various TFLiteSwift demo projects

2021 − "Golf Swing Clipper" App (iOS, On-device ML, Core ML, Vision framework, Pose Estimation) [App Store Link]

Develop a golf swing clipping app using MLKit's 3D Pose Estimation

• When the user imports a long golf swing video, it inference every frame to the skeleton asynchronously

• In the video player preview, the user can check the skeleton for the current video time

• Implement rule base feature to detect golf swing time range with the sequence of the skeleton that estimates with 3d pose estimation

• Implement interpolation for the sequence of keypoint and make skeleton movement smooth during the user's seeking control

2021 − Contribution to huggingface/transformers (NLP, GPT2, NER) [PR Link]

Make PR for GPT2ForTokenClassification class which configures by GPT2 for upstream and NER for downstream task

2020 − "PoseEstimation-TFLiteSwift" Repository (iOS, On-device ML, TFLiteSwift, Pose Estimation, Modularization) [Github Link]

Provide a sample source code for developers to easily use single and multi-person pose estimation models with Swift and TFLiteSwift

• Implement OpenPose paper's multi-person pose estimation post-processing with Swift by using a bipartite graph, line integral, non-maximum suppression, etc.

• Design extensible code architecture for supporting various shapes of outputs of pose estimation models(OpenPose, PoseNet, Hourglass, and CPM)

• Implement drawing heat map(PCM and PAF) on the screen for the model's output debugging

• Implement 3D pose estimation pre-processing like soft-argmax in Swift

• Implement to draw 3d key points on the screen with SceneKit and render the key points in real-time

2020 − "SemanticSegmentation-CoreML" Repository (iOS, On-device ML, Core ML, Metal, Semantic Segmentation) [Github Link]

Provide a sample source code for developers to easily use the semantic segmentation model with Swift and Core ML framework

• Implement to draw segmentation-map with CPU(CoreGraphics framework) and GPU(Metal framework), and compare each performance

2020 − "Capture& Paste" App (macOS, On-device ML, Core ML, Vision framework, OCR) [App Store Link]

Develop a simple utility macOS app using Text Recognition with Vision framework and open-source libraries

2019 − "Just Point It" App (iOS, Vision, On-device ML, ML Kit, Core ML, Models' Pipeline, Fingertip Pose Estimation, OCR, Post-processing Optimization) [App Store Link]

Design, develop and release the "Just Point It" app using vision-based 3 machine learning models and optimization algorithms

• Implement the pipeline of image data through pose estimation(Core ML), text detection(Vision framework), and text recognition(ML Kit) models

• Use the majority voting algorithm to improve the performance

2019 − "PoseEstimation-CoreML" Repository (iOS, On-device ML, Core ML, Pose Estimation, Pose Matching) [Github Link]

Provide a sample source code for developers to easily use pose estimation model with Swift and Core ML framework

• Implement converting logic from camera frame to pixel buffer for pre-processing and argmax for converting heat-map to joint location for post-processing

• Make pose matching algorithm which scores for comparing between target pose and inferred pose

• Implement to draw a heatmap (or heatmaps) on the screen for model's output debugging

2018 − "awesome-ml-demos-with-ios" Repository [Github Link]

Collect implementation of various pre/post-processing of each vision model on iOS

• My implementations included in:
  DepthPrediction-CoreML, ImageClassification-CoreML, ObjectDetection-CoreML, TextDetection-CoreML, TextRecognition-MLKit, FaceDetection-MLKit, etc.

• Measure the latency of the inference time and post-processing time with 3 ways on mobile device

• Compare the latencies between the iOS devices each model

Other Projects

• 2019 − ReHapp (iOS, Pose Estimation, Pose Matching) [Github Link]

• 2018 − Don't Be Turtle (iOS, Pose Estimation) [Github Link]

• 2016 − Roast Meister (iOS, iPhone&iPad) [App Store Link]

• 2014 − Hangul Clock (iOS, iPhone&iPad) [App Store Link]

• 2014 − Guitar Kit+ (iOS, Cocos2D Engine) [App Store Link]

• 2013 − Easi Calc (iOS) [App Store Link(free)] [App Store Link(Pro)] [App Store Link(Widget)]

Publications & Blog Posts & Presentations

• 2024.05 – Presented and co-hosted a Q&A session at Google Korea Campus Outreach (Kanwon University)

• 2023.09 – Presented and co-hosted a Q&A session at Google Korea Campus Outreach (Inha University)

• 2023.07 – Presented a session about "Easy on-device ML with MediaPipe" at AI Week 2023 | KOREA, a tech conference co-hosted with the Ministry of Science and ICT

• 2022.11 – Co-translated the Korean edition of "AI and Machine Learning for On-Device Development" with Chansung Park [yes24 link(kr)]

• 2021.08 – Published a blog post: "Find a restaurant with a good night view in Naver service (Vision, Image Classification, Semi-supervised learning, Hierarchical Classification)" [Blog Post Link(kr)]

• 2020.11 – Presented and co-hosted a Core ML Panel and delivered a lightning talk at Let's Swift 2020 Online [Video Link(kr)]

• 2020.09 – Published a Medium blog post: "Annotating 40K Images Without Losing Engagement Using ML" [Blog Post Link(kr)]

• 2020.08 – Presented "2020 What's New in Apple's Machine Learning" at the ApplicableML community [Slide Link(kr)]

• 2019.12 – Presented "Will my ML model work on the mobile app" at MODUCON 2019 Conference [Slide Lin(kr)], [Video Link(kr)]

• 2019.11 – Presented "Problems that machine learning can solve in iOS apps" at Let's Swift 2019 [Slide Link(kr)], [Video Link(kr)]

• 2019.08 – Presented "Machine Learning with Apple" as a special lecture at Dongguk University [Slide Link(kr)]

• 2019.05 – Published a Medium blog post: "Just Point It": Machine Learning on iOS with Pose Estimation + OCR using Core ML and ML Kit [Blog Post Link(en)]

• 2019.03 – Presented "'Just Point It', A dictionary that finds you when you point it out" at NAVER TechTalk [Slide Link(kr)], [Video Link(kr)]

• 2018.06 – Presented "Try Core ML" at the MotLabs community [Slide Link(kr)]

Community Activity

• 2020~Present − Applicable ML [Community Blog]

  • With the people who are interested in mobile application + ML, develop personal projects and share what we learn 

  • Participate as a community leader

• 2018~2019 − MoT Labs [Community Blog]

  • Participate as an iOS developer

Honor & Awards

• 2019 − 6th NAVER D2 CAMPUS FEST, 1st Place Award

• 2016 − Guitar Kit+ featured on App Store, "Learn to Play"

• 2015 − 2015 Startup Weekend Busan, Best Engineering Award

• 2015 − Guitar Kit+ featured on App Store, "New Year, New You", "Indoor leisure activities" and "Best Music"

• 2014 − Guitar Kit+ featured on App Store, "Weekly Best"