We tried 3 novel ways of modifying the transformer based DETR architecture for the visual grounding task. We took a model that had been trained on a object detection task with no prior interaction with text and fine-tuned it to produce acceptable results on visual grounding tasks within 20 epochs of training. We were able to improve our baseline architecture's mean Average Precision performance by 6 % in subsequent experiments by incorporating improvements.

We would like to note that while we were able to improve upon our baseline, our model performance is not comparable to the existing State of the Art models like MDETR. This is due to the following reasons.

• MDETR was pre-trained on the modulated detection task where the aim is to detect all objects that are referred to in the aligned text. This is a very similar task to visual grounding. Our model introduces the text aspect only during fine-tuning

• MDETR was pre-trained for 1 week using 32 v100 GPUs and finetuned with similar resources. We finetuned each experiment for 600 hours on a single T4 GPU. This is a massive difference in computing power and time for which the model was exposed to text features.

However, we would also like to add that the AP values reported by DETR are 42%. Our models, which have been fine-tuned from DETR and bootstrapped to include a second modality were able to give AP values close to 36%. These numbers are very comparable and are a very promising starting point.