[Future Work] NS-Driven Multimodal On-Device AI for Surgical and Embodied Robotics.
[Ongoing Work] NS-Driven Model–System Optimization for On-Device Video VLMs and VLAs.
[Extension] Memory- and Energy-Efficient Model Adaptation for On-Device AI
Cross-Layer Characterization, Model Selection, and Efficient On-Device AI Model Design.
Adaptive Resource Management for On-Device AI
Energy-efficient Runtime Management for Heterogeneous Mobile Systems
Building on ongoing work 1, I will apply the developed NS-driven model--system optimization methods to real-world applications, particularly surgical robotics, including dental and general surgical procedures, and broader embodied robotic systems. Leveraging my ongoing dental video classification research and proposed work on efficient surgical AI models, I will investigate efficient Video VLMs for real-time surgical scene understanding, anatomical structure and instrument recognition, and temporal procedure analysis, alongside VLAs for vision- and language-guided robotic manipulation and surgical assistance.
I will apply and refine the Pareto-efficient configurations from Future Work 1 for resource-constrained surgical and embodied robotic platforms, enabling efficient on-device fine-tuning and low-latency inference. This research will jointly address application-specific requirements for real-time responsiveness, robustness, and safety alongside accuracy, memory, latency, and energy efficiency. Through simulation, robotic testbeds, and expert-supervised validation, I aim to demonstrate practical deployment of resource-efficient multimodal AI in real-world robotic systems.
As an immediate extension of my ongoing research [arXiv 2026a][arXiv 2026c], I will establish a ongoing work on NSE, NSM, and NS#-driven efficient multimodal on-device AI design and optimization, specifically targeting video vision–language models (Video VLMs) for temporal video understanding and reasoning, and vision–language–action models (VLAs) for embodied perception and action generation. Rather than training these models from scratch, I will focus on resource-efficient adaptation of pretrained video models through fine-tuning and model–system co-optimization on embedded platforms. This research will jointly investigate model-side choices---including PEFT configurations studied in [arXiv 2026a], frame and token selection, and temporal computation---and system-side choices---including adaptive activation checkpointing developed in [arXiv 2026c], memory management, execution scheduling, and DVFS---alongside cross-layer optimizations such as mixed-precision computation and quantization. I will employ hierarchical HPO--NAS and advanced multi-objective search strategies, combining Bayesian optimization, adaptive search-space refinement, and local search to efficiently explore the joint model--system design space. Guided by NSE, NSM, and NS#, these methods will identify Pareto-efficient configurations that balance task accuracy, peak memory consumption, execution time, and energy.
My third research thrust extends efficient on-device AI from inference and model selection toward resource-efficient model adaptation. I developed energy- and memory-efficient PEFT methods for personalized SLMs, systematically evaluating adaptation strategies across Transformer- and Mamba-based models under constrained computing resources [arXiv 2026a]. I then extended this direction to vision models and VLMs by combining PEFT with adaptive activation checkpointing, enabling fine-tuning under explicit memory constraints on resource-constrained GPUs [arXiv 2026c]. Complementing these adaptation techniques, my GPU-DVFS work demonstrated that the energy cost of on-device fine-tuning can also be reduced through adaptive hardware operation [arXiv 2026b]. Across these studies, I used NSE, NSM, and NS# to characterize the tradeoffs among predictive performance, energy consumption, and peak memory during model adaptation. Together, this work establishes a foundation for memory- and energy-efficient model adaptation on resource-constrained platforms and motivates my future research on joint model--system optimization.
My second research thrust addresses cross-layer complexity by characterizing how workloads, AI models, software frameworks, runtime choices, and hardware platforms jointly affect efficiency, and by using this understanding to guide model selection and design.
For cross-layer characterization, from a system-level perspective, I developed a CNN inference microbenchmark that compares direct and unrolling-based convolution strategies and adaptively selects kernels for individual CNN layers to reduce GPU inference time without accuracy loss [SMC 2022]. I then analyzed performance differences across cuBLAS, cuDNN, and TensorRT using GPU kernel profiling and interpretable ML models [SAC 2023], and extended this analysis by characterizing deep-learning libraries across CNN and ViT workloads on Jetson TX2 and RTX 3060 platforms [IEEE Access 2026a].
Moreover, from an application perspective, I also examined how workload and data characteristics influence learning behavior and model choice, including reinforcement-learning variability [ICTC 2021], brightness and class imbalance in autonomous-driving workloads [ICONIP 2023], and data-distribution effects in offline reinforcement learning [BigComp 2024].
For model selection, my research addresses two complementary perspectives: general-purpose model and system efficiency, and application-specific requirements. From a system perspective, I first investigated tradeoffs between predictive accuracy and interpretability to develop quantitative criteria for ML model selection [ICITEE 2021], and examined the limitations of premodel building for adaptive CNN model selection using ImageNet workloads [CTISC 2022]. I further extended model selection to hardware-aware efficiency by developing a two-stage framework for energy-efficient ViT inference that combines device-independent model screening with real-device measurements of accuracy, latency, and energy across GPU platforms [arXiv 2025a].
From an application perspective, particularly in healthcare, I developed PiSO, a pipelined framework that jointly selects preprocessing strategies and deep-learning models for arterial blood-pressure estimation PiSO: IEEE Access 2026b]. In related work, I contributed to an AI-based non-invasive continuous blood-pressure monitoring system integrating ECG sensing and deep-learning-based prediction [ASR 2026]. The Healthcare Monitoring projects includes a collaboration work with the Biosensors Lab (PI: Taeil Kim) at Baylor University.)
For model design, I compared mimic learning and feed-forward interpretable deep models using accuracy and simulatability-based metrics [BigComp 2023b], characterized how patch size and model size influence the accuracy and efficiency of on-device Vision Transformers [IEEE Access 2025], and developed hyperparameter optimization for efficient multimodal Transformers [arXiv 2025b]. Together, these studies establish a characterization-to-selection-and-design methodology for developing and deploying efficient AI models across diverse workloads and embedded computing platforms.
Building on the ML-enhanced CPU–GPU DVFS governor introduced in my PhD work [ML-Gov: ESTIMedia 2017], I extended this integration of embedded systems and machine learning toward systematic model selection and interpretable runtime decision-making [ACM TECS 2021]. In this journal extension, I investigated interpretable predictive models and workload descriptors, including simulatability operation counts (SOC), to capture workload-dependent power–performance behavior and dynamically scale CPU and GPU frequencies for improved energy efficiency. I subsequently explored offline reinforcement-learning-oriented objective-function design for dynamic frequency scaling of CNN workloads [BigComp 2023a] and integrated CNN workload characterization with CPU–GPU DVFS governors on embedded platforms [IEEE ESL 2023]. More recently, I extended adaptive runtime resource management to on-device AI training by developing GPU-DVFS techniques that reduce the energy cost of fine-tuning small language models (SLMs) on resource-constrained embedded platforms [arXiv 2026b]. Collectively, this thrust extends my earlier CPU–GPU power-management research toward adaptive resource management for modern AI workloads, with a focus on improving energy efficiency while maintaining model performance.
As a graduate student at UCI, I focused on energy-efficient runtime management for emerging heterogeneous mobile GPU systems, particularly on coordinating CPU, GPU, and memory resources under dynamic workloads. 1) First, I characterized mobile graphics workloads from a quality-aware perspective to understand the relationships among performance, visual quality, and energy consumption [ESTIMedia 2014]. 2) Second, I developed two complementary runtime power-management approaches: cooperative CPU–GPU DVFS, including memory-aware control [ESTIMedia 2015; DAEM 2018], and cooperative and hierarchical frequency capping for energy-efficient mobile gaming [Co-Cap: SAC 2016; HiCAP: ISLPED 2016; ACM TECS 2018]. 3) Finally, I extended these adaptive runtime techniques with a machine-learning (ML) enhanced CPU–GPU DVFS governor using model trees that learned workload-dependent power–performance behavior to improve runtime decision-making [ML-Gov: ESTIMedia 2017]. Collectively, this work established my foundation in adaptive resource-aware computing and led naturally to my later research on efficient on-device AI.
Block 7e.443, 53 Kabanbay Batyr Ave. Astana 010000, Kazakhstan
E-mail: jurn.park@nu.edu.kz Tel: +7-7172-70-5759,
ESML lab website (lab location: NU C4.406)
Copyright©2024 ESML Research Group.