Idea
Idea
Embedded Files
Similar to MDETR, our idea also builds on DETR to use it for the visual grounding task rather than the object detection task, though we explore other avenues to do this. We explore 3 different methods as listed below.
Experiment 1: Cross Attention Module
Experiment 1: Cross Attention Module
Our initial approach to modify the DETR architecture for the visual grounding task involves replacing the self attention transformer with a form of cross attention where we replace the query with the text embeddings obtained from RoBERTa. We start we the pretrained DETR and keep all weights frozen except that of the encoder. We fine tune the encoder with the new Query input. We treat this as our baseline. We believe a cross modality attention module can help identify the sections of the image relevant to the provided prompt.
Experiment 2: Stacked Cross Attention
Experiment 2: Stacked Cross Attention
Building on idea 1, we believe that a few layers of self-attention for the image features would be a more useful source of information. Hence, we propose stacking self-attention layers with our cross-attention module from approach 1. Our model includes 3 layers of self-attention followed by 3 layers of cross attention. The rest of the downstream architecture remains the same as with DETR.
Experiment 3: Text Embedding Based Decoder Input
Experiment 3: Text Embedding Based Decoder Input
Our final experiment involves the input to the decoder. In DETR, the decoder takes as input, a set of zeros along with the "learnable parameter" object query as the positional embedding. We believe that providing the decoder with information about the prompt may help the DETR architecture accomplish the visual grounding task. In our experiment, we feed in text embeddings obtained from RoBERTa as the input to the Decoder along with the object query as the positional embedding. In this setup, we keep all the weights of the pretrained model frozen except the decoder.
Page updated
Report abuse