ONE MORE GLANCE
WITH SHARP EYES:
Rethinking Lightweight Captioning as a Practical Visual Specialist
What is it?
Humans first take in the overall scene, then glance at specific regions to notice finer details. Our Sharp-Eyed Refinement framework mimics this human tendency, allowing the captioning specialist to revise and improve initial descriptions.
Junha Song1 Yongsik Jo2 So Yeon Min3 Quanting Xie3
Taehwan Kim2 Yonatan Bisk3 Jaegul Choo1
1KAIST 2UNIST 3Carnegie Mellon University
🧚 Main Story
1. Image captioning serves as the cornerstone of diverse applications such as assistive devices.
2. However, running multimodal large language models for this task is challenging for edge devices.
3. We highlight the practical importance of lightweight captioning.
What We Have Demonstrated?
➡️ Swipe to view more results ➡️
🫢 Factual image captioning does not significantly require the complex reasoning abilities of LLMs and can be handled by a lightweight language model.
😲 Single-pass captioning can lead specialists to adopt broad attention, neglecting critical visual regions necessary.
🧐 CLIP’s final-layer features provide only coarse-grained visual information, limiting their suitability for fine-grained captioning tasks.
Novel Caption Refinement Framework:
Sharp-Eyed Refinement
The word “sharp-eyed” means having keen eyesight or being observant. (▻.◅')
What is it? Humans first take in the overall scene, then glance at specific regions to notice finer details. Our Sharp-Eyed Refinement framework mimics this human tendency, allowing the captioning specialist to revise and improve initial descriptions
Our novel insight: 🥳 Encouraging attention to key regions via the initial caption, along with multi-level utilization of a single vision encoder, can be an effective and efficient strategy.
Why novel? To the best of our knowledge, this is the first work to apply multimodal refinement that jointly utilizes visual features and the model’s initial textual output.
Broad Comparisons
This is the first paper to demonstrate the amazing capability of a lightweight captioning model beyond single-sentence captioning, including not only detailed captioning but also long-form VideoQA.
Unlike traditional works, we compared not only existing small specialist models but also large multimodal models, demonstrating their potential as strong visual specialists.
Positive Influences
Industry: Our model can be easily applied in industrial scenarios such as assistance devices for the visually impaired or navigating robots. Our model runs successfully on actual edge devices like Jetson Nano.
Academy: For researchers with limited access to large-scale GPUs who are unable to operate LLMs, our factual image captioning model offers a promising baseline.
We evaluate our lightweight captioner on both single-sentence and detailed captioning tasks.
🔥 It uses 93% fewer parameters and offers 97% faster inference than LLaVA-1.5-7B.🔥
Abstract
Image captioning is fundamental for applications like video instruction systems and exploration robots, yet deploying such models on local devices is challenging due to the high computational demands of multimodal large language models (MLLMs).
To address this, we first explore lightweight captioning by implementing a specialist based on a 125M-parameter language model, 56 times smaller than LLaMA-7B, and evaluating its performance on both single-sentence and detailed captioning tasks. Surprisingly, we find that our model can achieve performance comparable to large multimodal generalists, suggesting its potential to serve as a strong visual specialist for on-device applications.
While promising, our model also exhibits a limitation: like other MLLMs, it suffers from visual blindness, occasionally resulting in semantic captioning errors. We carry out toy experiments and investigate the underlying causes, where we observe that the problems arise from ineffective attention mechanisms and limited visual representations.
To alleviate them, we develop a novel captioning framework, Sharp-Eyed Refinement, which enhances caption quality through improved visual grounding. At its core, our DeepLens extracts detailed visual representations by concentrating on informative regions identified during the initial glance. Our experiments confirm both the advantages of our specialist over prior small captioning models and large generalists and the effectiveness of our framework.
Experimental Results
1) Lightweight Captioning
We focused on minimizing reliance on LLMs, as they contribute to the majority of computational overhead in MLLMs. To this end, we simply build our captioning specialist by replacing the LLaMA-7B used in LLaVA construction with OPT-125M, a language model 56 times smaller.
We anticipated that our specialist would struggle with this challenging task, given that OPT-125M possesses limited language processing capabilities.
However, our specialist performs surprisingly well even on detailed captioning tasks.
2) Sharp-Eyes Refinement
Our framework yields both quantitative and qualitative improvements in caption quality, as shown in the figure and table below.
Its effectiveness remains constrained by the limitations of single-pass generation.
Since captioners are expected to serve as foundational components in downstream applications, it is important to verify their utility in practical settings. To this end, we evaluate various captioners on the Long-Range Video Question Answering task. Our specialist achieves competitive accuracy with significantly fewer parameters and faster inference than large-scale generalists.
Cite
@misc{song2025glancesharpeyesrethinking,
title={One More Glance with Sharp Eyes: Rethinking Lightweight Captioning as a Practical Visual Specialist},
author={Junha Song and Yongsik Jo and So Yeon Min and Quanting Xie and Taehwan Kim and Yonatan Bisk and Jaegul Choo},
year={2025},
eprint={2508.21451},
archivePrefix={arXiv},
}