As we can see from the graphs above, of the 3 experiments, stacked attention performed the best. It makes sense that the stacked attention performs better that the pure cross attention baseline as it provides all the benefit of the baseline module, while also having attention within the image features prior to the cross attention layers. It was interesting to see that experiment 3's output was comparable to experiment 2 as in experiment 3 the only source of text input is the RoBERTa input to the decoder.

Due to time and compute constraints, each of these models were trained for only 20 epochs. Transformers generally take a long time to train, and our experiments would potentially give better results if trained for longer durations.