Moon Ye-Bin

Model Discovery / VLM & LLM / Agentic System

• Interpreting time-series data as visual graphs with VLMs yields higher model discovery accuracy than interpreting it as text with LLMs. By leveraging VLM-based agents, the system visualizes and analyzes the given time-series data to propose model candidates, then iteratively evaluates and refines them to identify the most suitable model.

Data Augmentation / Diffusion Models

• SYNAuG augments imbalanced training data with synthetic images to balance the data distribution. Although there exists a domain gap between synthetic and real images, incorporating synthetic data is beneficial in data-imbalanced settings when at least 5–10 real samples are available. In tasks involving long-tail recognition, model fairness, and spurious correlation, we demonstrate that SYNAuG outperforms algorithmic approaches trained only on real data, suggesting the importance of controlling imbalance from a data perspective.

• [Workshop] ICCVw 2023 (MMFM: What is Next in Multimodal Foundation Models?)

Model Evaluation / VLM & LLM / SFT

• VLM’s Eye Examination investigates how a VLM perceives images, specifically focusing on key elements of visual recognition, from primitive color and shape to semantic levels. Models that fail to understand the exam format undergo supervised fine-tuning before evaluation. The color exam reveals that VLMs are less sensitive to the green spectrum. Furthermore, we observe that the LLM component, which serves as the cognitive core of the VLM, influences shape sensitivity and patch-wise semantic discrimination.

Model Evaluation / VLM & LLM

• BEAF evaluates whether a VLM correctly understands a scene by observing its response changes to the same question when presented with an original image versus a manipulated version from which an object has been removed. Based on the proposed change-aware metrics, we demonstrate that answers previously considered non-hallucinatory by existing benchmarks may actually be guesses made without referencing the image content. This finding suggests that observing changes along the vision-axis is crucial for accurately assessing VLM hallucination.

Data Augmentation / VLM & LLM 

• TextManiA augments visual features by incorporating attribute vectors from the text embedding space. Unlike conventional noise injection techniques, TextManiA makes augmentation understandable and semantically meaningful. Surprisingly, we demonstrate that visual features can be effectively augmented not only using aligned text-visual spaces like CLIP, but also by leveraging embeddings from independent LLMs such as GPT-2 and BERT.

• [Workshop] CVPRw 2023 (WFM: Workshop on Foundation Models)

• [Workshop] ICCVw 2023 (MMFM: What is Next in Multimodal Foundation Models?)

Few-shot Segmentation / Markov Random Field / Parameterization

• ENInst includes sub-task enhancement methods: MRF-based instance-wise refinement to improve pixel localization quality, and novel classifier composition that parameterizes them with base classifiers and Gaussian random vectors to improve classification accuracy, based on performance bottleneck analyses.

• [Conference] IPIU 2022, Best Paper Award

Scene Reconstruction / HDR

• HDR-Plenoxels learns the plenoptic function of the 3D scene from a comprehensive understanding of 3D information, physical radiance field, and varying camera settings inherent in 2D LDR images.

• [Workshop] CVPRw 2023 (3DMV: Learning 3D with Multi-View Supervision)

Federated Learning / Parameterization

• FedPara re-parameterizes layer weights using a low-rank Hadamard product, allowing larger capacity with lower communication costs than the model with the original layers in a federated learning (FL) setting. pFedPara, a personalized FL application based on FedPara, splits layer weights into global and local parameters and shows more robust results than competing methods.