Ting Lei1, Shaofeng Yin1, Qingchao Chen2, Yuxin Peng1, Yang Liu1
1 Wangxuan Institute of Computer Technology, Peking University
2National Institute of Health Data Science, Peking University
Open Vocabulary Human-Object Interaction (HOI) detection aims to detect interactions between humans and objects while generalizing to novel interaction classes beyond the training set. Current methods often rely on Vision and Language Models (VLMs) but face challenges due to suboptimal image encoders, as image-level pre-training does not align well with the fine-grained region-level interaction detection required for HOI. Additionally, effectively encoding textual descriptions of visual appearances remains difficult, limiting the model’s ability to capture detailed HOI relationships. To address these issues, we propose Interaction-aware Prompting with Concept Calibration (INP-CC), an end-to-end open-vocabulary HOI detector that integrates interaction-aware prompts and concept calibration. Specifically, we propose an interaction-aware prompt generator that dynamically generates a compact set of prompts based on the input scene, enabling selective sharing among similar interactions. This approach directs the model’s attention to key interaction patterns rather than generic image-level semantics, enhancing HOI detection. Furthermore, we refine HOI concept representations through language model-guided calibration, which helps distinguish diverse HOI concepts by leveraging structured semantic knowledge. A negative sampling strategy is also employed to improve inter-modal similarity modeling, enabling the model to better differentiate visually similar but semantically distinct actions. Extensive experimental results demonstrate that INP-CC significantly outperforms state-of-the-art models on the SWIG-HOI and HICO-DET datasets.
Given an input image, we first extract the spatial feature map using a pre-trained CLIP image encoder, augmented with our interaction-aware prompts. These prompts consist of a common prompt, shared across all scenes, and individualized prompts that are tailored to specific interaction patterns. We then adaptively select and refine these prompts based on the input image. To further enhance HOI recognition, we use embeddings calibrated via language models to model intra-modal relationships, producing K distinct clusters. During training, we sample negative categories from the clusters that overlap with the positive categories in the current batch, improving inter-modal similarity modeling. This approach helps create a more structured and discriminative semantic space for HOI concepts.
@inproceedings{INP-CC_ICCV2025,
title={Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration},
author={Ting Lei, Shaofeng Yin, Qingchao Chen, Yuxin Peng, Yang Liu},
year={2025},
booktitle={ICCV},
organization={IEEE}
}