Last updated: May 20, 2025
Focus on the abstracts, introductions, and conclusion sections of technical papers for high-level understanding.
Use summary articles and videos to supplement your reading of original research.
Don’t hesitate to revisit foundational concepts like attention and self-attention, as these are central to all modern generative AI.
This curated list is designed for undergraduate computer science students with limited math backgrounds. It tracks the evolution of generative AI from the seminal "Attention Is All You Need" paper through major milestones up to the latest architectures, emphasizing accessible papers and summaries.
1. The Foundation: The Transformer
"Attention Is All You Need" (Vaswani et al., 2017)
Introduced the Transformer architecture, replacing recurrence with attention mechanisms, enabling parallelization and scalability in language models.
Attention is All You Need, https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf
2. Early Transformer Applications and Improvements
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018)
Introduced bidirectional pre-training, making Transformers excel at understanding context in text (encoder-only model).
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, https://arxiv.org/abs/1810.04805
GPT Series (OpenAI, 2018–2023)
GPT-2 (2019): "Language Models are Unsupervised Multitask Learners" by Radford et al., OpenAI, 2019
Showed that scaling decoder-only Transformers could generate coherent, long-form text.
Language Models are Unsupervised Multitask Learners, https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
GPT-3 (2020): "Language Models are Few-Shot Learners" by Brown et al., OpenAI, 2020
Demonstrated few-shot learning and task generalization through massive scaling.
Language Models are Few-Shot Learners, https://arxiv.org/abs/2005.14165
3. Expanding Beyond Text: Multimodal and Vision Models
CLIP: Connecting Text and Images (Radford et al., 2021)
Combined vision and language using Transformers, enabling models to understand images and text together.
CLIP: Connecting text and images, https://openai.com/index/clip/
Vision Transformer (ViT, Dosovitskiy et al., 2020)
Applied Transformer architectures to image classification, showing their versatility beyond text.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, https://arxiv.org/abs/2010.11929
4. Efficiency and Scaling Innovations
Switch Transformer (Fedus et al., 2021)
Introduced sparse models, improving efficiency by activating only parts of the model per input.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, https://arxiv.org/abs/2101.03961
LLAMA 2 (Meta, 2023)
Open-source, efficient large language model, advancing dialogue systems and accessibility.
Llama 2: Open Foundation and Fine-Tuned Chat Models, https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/
5. Modern Generative AI: Multimodal and Large-Scale Models
DALL-E (Ramesh et al., 2021)
Generated images from text prompts, showcasing generative capabilities of Transformers in creative tasks.
Zero-Shot Text-to-Image Generation, https://arxiv.org/abs/2102.12092
Stable Diffusion 3 and Sora (2024)
State-of-the-art image and video generators using Transformer-based architectures for high-quality, controllable outputs.
GPT-4 (OpenAI, 2023)
Multimodal capabilities, improved reasoning, and reduced transparency, representing the current frontier of generative AI.
GPT-4 Technical Report, https://arxiv.org/abs/2303.08774
Wikipedia: "Attention Is All You Need"
Provides a non-technical summary of the paper’s impact and the Transformer’s role in modern AI.
Attention Is All You Need, https://en.wikipedia.org/wiki/Attention_Is_All_You_Need
VisionX Blog: "A Comprehensive Exploration of Generative AI Architecture"
Offers a broad, approachable overview of generative AI’s evolution, including early models, GANs, VAEs, and the rise of Transformers.
A Comprehensive Exploration of Generative AI Architecture, https://visionx.io/blog/generative-ai-architecture/
LinkedIn: "A Quick Summary of Research Papers regarding Generative AI"
A bit old, but summarizes the key papers and their contributions to generative AI, useful for students seeking a high-level understanding.
https://www.linkedin.com/pulse/quick-summary-research-papers-regarding-generative-ai-soumava-dey/
YouTube: "The 7 AI Papers bridging 2017 to 2024 – Transformers Everywhere"
A video guide covering the most influential papers in the field, ideal for visual learners.