• A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping [arXiv]

Houjian Yu, Mingen Li, Alireza Rezazadeh, Yang Yang and Changhyun Choi

• Prepared to release ETRG-B model for Referring Grasp Affordance task. Stay tuned!

• [02/07/2025] Github Repo: ETRG-A-RGS model implementation has been released.

• [02/05/2025] Github Repo: ETOG model implementation for Referring Expression Segmentation task has been released.

• [01/27/2025] Our paper has been accepted for ICRA 2025! See you in Atlanta!

The language-guided robot grasping task requires a robot agent to integrate multimodal information from both visual and linguistic inputs to predict actions for target-driven grasping. While recent approaches utilizing Multimodal Large Language Models (MLLMs) have shown promising results, their extensive computation and data demands limit the feasibility of local deployment and customization. To address this, we propose a novel CLIP-based multimodal parameter-efficient tuning (PET) framework designed for three language-guided object grounding and grasping tasks: (1) Referring Expression Segmentation (RES), (2) Referring Grasp Synthesis (RGS), and (3) Referring Grasp Affordance (RGA). Our approach introduces two key innovations: a bi-directional vision-language adapter that aligns multimodal inputs for pixel-level language understanding and a depth fusion branch that incorporates geometric cues to facilitate robot grasping predictions. Experiment results demonstrate superior performance in the RES object grounding task compared with existing CLIP-based full-model tuning or PET approaches. In the RGS and RGA tasks, our model not only effectively interprets object attributes based on simple language descriptions but also shows strong potential for comprehending complex spatial reasoning scenarios, such as multiple identical objects present in the workspace.

Efficient-Tuning pipeline for language-guided object grounding and robot grasping tasks. We propose Efficient-Tuning Object Grounding (ETOG) for RES ask, Efficient-Tuning Robot Grasping type-A (ETRG-A) for RGS task, and type-B (ETRG-B) for RGA task. Our framework with minor modifications is able to solve the three tasks.