Research

While generative AI has significantly advanced 3D scene generation and layout synthesis, traditional text-only prompting creates a severe bottleneck: users struggle to describe spatial intentions—such as precise object references, alignments, and spatial relationships—through linear, descriptive text alone. This spatial-to-text gap leads to ambiguous descriptions, frequent trial-and-error prompting, and unnecessary cognitive burden. 

To address this challenge, this project explores a direct manipulation interface integrated with a 3D viewport that automatically translates user spatial interactions (e.g., selecting, dragging, and arranging 3D entities) into structured prompt contexts for vision-language models. By bridging the gap between spatial manipulation and generative AI inputs, we aim to minimize prompting friction, establish intuitive spatial interaction mechanisms, and enhance the consistency and controllability of AI-assisted 3D creation workflows. 

Associated Studies

Virtual production workflows increasingly demand seamless integration between real-time rendering environments, physical studio spaces, and complex digital assets. However, current pipelines face significant operational bottlenecks: filmmakers and virtual creators still heavily rely on manual calibration, fragmented asset management, and high-latency adjustments during live on-set production. These challenges elevate production overhead and severely limit rapid prototyping in dynamic shooting scenarios.

Under the AI Star Fellowship Support Program, this project investigates intelligent computing and generative interaction techniques tailored for next-generation virtual production environments. By integrating spatial reasoning, AI-driven asset adaptation, and intuitive human-in-the-loop interfaces into real-time production stages, we aim to streamline virtual filmmaking workflows, enhance real-time scene control, and empower creators to intuitively realize their creative visions on set.

Associated Studies

Industrial manufacturing environments, such as smart factories and manufacturing execution systems (MES), rely on complex operational interfaces that must continuously adapt to evolving assembly lines, equipment, and production parameters. However, traditional industrial UIs remain rigid, requiring extensive manual reconfiguration and domain-specific engineering whenever workflows change. This creates a critical bottleneck in agile manufacturing, impeding real-time collaboration between human operators, autonomous machinery, and physical AI systems on the shop floor.

As part of this 5-year initiative, this project investigates AI-driven UI generation and dynamic interface reconfiguration tailored for modular SDF-OCS (Software-Defined Factory Operation & Control System) environments. By developing standardized physical AI software layers, we synthesize domain-aware interfaces on the fly—translating real-time sensory data, machine states, and production constraints into adaptive, user-centered control panels. Our goal is to establish interoperable software standards and next-generation industrial interface architectures that enable seamless, intuitive human-AI collaboration across physical manufacturing workflows.

Associated Studies