Doyoung Gwak
Software Engineer (Google)
E-mail: tucan.dev[at]gmail.com
Github: https://github.com/tucan9389
Linked In: https://www.linkedin.com/in/tucan9389
I am currently working on optimizing models by applying quantization techniques at Google in South Korea. This area is really interesting to me because it helps reduce the costs associated with machine learning and improves user experiences.
I've also been deeply interested in on-device machine learning (ODML) for a long time. This passion has been a significant influence throughout my career. I started out in iOS app development and later expanded my skills to include server-side machine learning engineering.
Work Experience
2022.09~Present Google LLC
Optimize Server-side models via quantization techniques (Python, C++, and MLIR)
Develop the Quantization Debugging feature for Server-Side models [commit]
Optimize ODML(On-device ML) models via quantization techniques (Python, C++, and MLIR)
Reinforce and maintain PTQ, QAT conversion pipeline for TensorFlow Lite
Improve the performance of TensorFlow Lite converter (x2 reduction of large model conversion)
Maintain the TensorFlow Lite converter, working with both MLIR manipulation level and FlatBuffer manipulation level [commits]
Fundamental effort (MLIR)
Support MLIR for netron.app model visualization (PR)
2019.09~2022.08 NAVER corp.
Develop and apply ML in real service in NAVER, like LINE CONOMI in Japan
Restaurant Atmosphere Image Classification (Vision, ML, Image Classification, Semi-supervised Learning, Pytorch)
Make a POC classifier with a sample dataset with teachable machine
Boost 4x times speed of building image classification annotation by pre-annotating with PoC classifier
Make a baseline code for image classification, train the own dataset, and do hyperparameter tuning
Image Rotation Detector in Server (Vision, ML, Image Classification, TensorFlow)
Collect and refine the dataset by using ML (using teachable machine's prototype model)
Train with models provided by Keras's applications and serve the trained model with Flask and gunicorn
Implement a Kafka consumer and ad-hoc batch to use the API
Receipt Image Edge Detector on iOS (Vision, ML, On-device ML, Pose Estimation, TensorFlow, TensorFlowLite)
Refine the receipt's 4 key points annotated dataset with ML (annotate and infer iteratively) with mobile annotation tool, which I made
Train the model with tf2-mobile-pose-estimation's models
Convert the TF2 model to the TFLite model and make a prototype application using the model
Show the heat-map output of the model on the mobile device
Recognize a receipt in a camera, whether it could be automatically captured or not (pre-processing of OCR request)
[intern] Food Image Detector Album on iOS (Vision, ML, On-device ML, Binary Image Classification, Create ML, Core ML)
Train a food or not classifier with an open dataset using Create ML
By using the model, implement a photo album that only shows food images
Implement user interactive inference batch in the background without UI blocking
Education
2012−2020: BS, Computer Science Engineering, Pusan National University, Pusan, Republic of Korea. GPA 3.48
Personal Projects
2021 − "TFLiteSwift-Vision" Repository (iOS, On-device ML, TFLiteSwift, Modularization, CocoaPods) [Github Link]
Provide a vision domain specific-library that implements pre/post-processing of vision task
Implement various preprocessing functions for vision tasks
Make use-cases in various TFLiteSwift demo projects
2021 − "Golf Swing Clipper" App (iOS, On-device ML, Core ML, Vision framework, Pose Estimation) [App Store Link]
Develop a golf swing clipping app using MLKit's 3D Pose Estimation
When the user imports a long golf swing video, it inference every frame to the skeleton asynchronously
In the video player preview, the user can check the skeleton for the current video time
Implement rule base feature to detect golf swing time range with the sequence of the skeleton that estimates with 3d pose estimation
Implement interpolation for the sequence of keypoint and make skeleton movement smooth during the user's seeking control
2021 − Contribution to huggingface/transformers (NLP, GPT2, NER) [PR Link]
Make PR for GPT2ForTokenClassification class which configures by GPT2 for upstream and NER for downstream task
2020 − "PoseEstimation-TFLiteSwift" Repository (iOS, On-device ML, TFLiteSwift, Pose Estimation, Modularization) [Github Link]
Provide a sample source code for developers to easily use single and multi-person pose estimation models with Swift and TFLiteSwift
Implement OpenPose paper's multi-person pose estimation post-processing with Swift by using a bipartite graph, line integral, non-maximum suppression, etc.
Design extensible code architecture for supporting various shapes of outputs of pose estimation models(OpenPose, PoseNet, Hourglass, and CPM)
Implement drawing heat map(PCM and PAF) on the screen for the model's output debugging
Implement 3D pose estimation pre-processing like soft-argmax in Swift
Implement to draw 3d key points on the screen with SceneKit and render the key points in real-time
2020 − "SemanticSegmentation-CoreML" Repository (iOS, On-device ML, Core ML, Metal, Semantic Segmentation) [Github Link]
Provide a sample source code for developers to easily use the semantic segmentation model with Swift and Core ML framework
Implement to draw segmentation-map with CPU(CoreGraphics framework) and GPU(Metal framework), and compare each performance
2020 − "Capture& Paste" App (macOS, On-device ML, Core ML, Vision framework, OCR) [App Store Link]
Develop a simple utility macOS app using Text Recognition with Vision framework and open-source libraries
2019 − "Just Point It" App (iOS, Vision, On-device ML, ML Kit, Core ML, Models' Pipeline, Fingertip Pose Estimation, OCR, Post-processing Optimization) [App Store Link]
Design, develop and release the "Just Point It" app using vision-based 3 machine learning models and optimization algorithms
Implement the pipeline of image data through pose estimation(Core ML), text detection(Vision framework), and text recognition(ML Kit) models
Use the majority voting algorithm to improve the performance
2019 − "PoseEstimation-CoreML" Repository (iOS, On-device ML, Core ML, Pose Estimation, Pose Matching) [Github Link]
Provide a sample source code for developers to easily use pose estimation model with Swift and Core ML framework
Implement converting logic from camera frame to pixel buffer for pre-processing and argmax for converting heat-map to joint location for post-processing
Make pose matching algorithm which scores for comparing between target pose and inferred pose
Implement to draw a heatmap (or heatmaps) on the screen for model's output debugging
2018 − "awesome-ml-demos-with-ios" Repository [Github Link]
Collect implementation of various pre/post-processing of each vision model on iOS
My implementations included in:
DepthPrediction-CoreML, ImageClassification-CoreML, ObjectDetection-CoreML, TextDetection-CoreML, TextRecognition-MLKit, FaceDetection-MLKit, etc.Measure the latency of the inference time and post-processing time with 3 ways on mobile device
Compare the latencies between the iOS devices each model
Other Projects
2019 − ReHapp (iOS, Pose Estimation, Pose Matching) [Github Link]
2018 − Don't Be Turtle (iOS, Pose Estimation) [Github Link]
2016 − Roast Meister (iOS, iPhone&iPad) [App Store Link]
2014 − Hangul Clock (iOS, iPhone&iPad) [App Store Link]
2014 − Guitar Kit+ (iOS, Cocos2D Engine) [App Store Link]
2013 − Easi Calc (iOS) [App Store Link(free)] [App Store Link(Pro)] [App Store Link(Widget)]
Publications & Blog Posts & Presentations
2022.11 − Korean translation book of "AI and Machine Learning for On-Device Development", with co-translator Chansung Park [yes24 link(kr)]
2021.08 − Find a restaurant with a good night view in Naver service (Vision, Image Classification, Semi-supervised learning, Hierarchical Classification) [Blog Post Link(kr)]
2020.11 − Participate in Core ML Panel, Let's Swift 2020 Online Lightning Talk [Video Link(kr)]
2020.09 − Annotate 40k images while you don't get bored (with ML), Medium [Blog Post Link(kr)]
2020.08 − 2020 What's New in Apple's Machine Learning, presentation in ApplicableML community [Slide Link(kr)]
2019.12 − Will my ML model work on the mobile app, MODUCON 2019 Conference [Slide Lin(kr)], [Video Link(kr)]
2019.11 − Problems that machine learning can solve in iOS apps, Let's Swift 2019[Slide Link(kr)], [Video Link(kr)]
2019.08 − Machine Learning with Apple, special lecture presentation in Dongguk University [Slide Link(kr)]
2019.05 − "Just Point It": Machine Learning on iOS with Pose Estimation + OCR using Core ML and ML Kit, Medium [Blog Post Link(en)]
2019.03 − "Just Point It", A dictionary that finds you when you point it out, NAVER TechTalk [Slide Link(kr)], [Video Link(kr)]
2018.06 − Core ML 맛보기, MotLabs community [Slide Link(kr)]
Community Activity
2020~Present − Applicable ML [Community Blog]
With the people who are interested in mobile application + ML, develop personal projects and share what we learn
Participate as a community leader
2018~2019 − MoT Labs [Community Blog]
Participate as an iOS developer
Honor & Awards
2019 − 6th NAVER D2 CAMPUS FEST, 1st Place Award
2016 − Guitar Kit+ featured on App Store, "Learn to Play"
2015 − 2015 Startup Weekend Busan, Best Engineering Award
2015 − Guitar Kit+ featured on App Store, "New Year, New You", "Indoor leisure activities" and "Best Music"
2014 − Guitar Kit+ featured on App Store, "Weekly Best"