Readings & Resources

Related Courses

Berkeley CS 294

Stanford CS 224N

Stanford CS 324

Waterloo CS 886

Benchmarks

Knowledge

1. MMLU: https://github.com/hendrycks/test?tab=readme-ov-file

2. SummEdits: https://github.com/salesforce/factualNLG

Reasoning Capability

1. ARC Prize: https://arcprize.org/ 

2. Agent Bench: https://github.com/THUDM/AgentBench

3. SWE Bench: https://github.com/princeton-nlp/SWE-bench

4. LeanDojo: https://leandojo.org/

5. MATH: https://github.com/hendrycks/math/

6. HumanEval: https://github.com/openai/human-eval

7. Chain-of-Thought Hub: https://github.com/FranxYao/chain-of-thought-hub

8. WebArena: https://arxiv.org/abs/2307.13854 

9. BrowsComp: https://openai.com/index/browsecomp/ 

10. GAIA: https://arxiv.org/abs/2311.12983 

11. TravelPlanner: https://osu-nlp-group.github.io/TravelPlanner/ 

12. SciInstruct: https://arxiv.org/abs/2401.07950 

Alignment

1. SafetyBench: https://github.com/thu-coai/SafetyBench

Benchmark Surveys

1. MetaTool: https://arxiv.org/pdf/2310.03128.pdf

2. LLM-eval-survey: https://github.com/MLGroupJLU/LLM-eval-survey

Multi-Modal

1. CURIE: https://github.com/google/curie

2. LAB-Bench: https://github.com/Future-House/LAB-Bench

3. MicroVQA: https://jmhb0.github.io/microvqa/

Scaling Laws

1. CAMEL: https://github.com/camel-ai/camel

References

(To be updated.)
(Note: some papers belong in more than one category)

LLM Capabilities & Eval

1. Are Emergent Abilities of Large Language Models a Mirage?

2. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

3. Large Language Models Cannot Self-Correct Reasoning Yet 

Reasoning (Prompt)

1. Self-Discover: Large Language Models Self-Compose Reasoning Structures

2. Take a step back: evoking reasoning via abstraction in large language models

3. Measuring and Improving the Faithfulness of Model-Generated Reasoning

4. Chain-of-Thought Reasoning Without Prompting

5. Retrieval-Augmented Planning

Understanding Reasoning

1. Why think step by step? Reasoning emerges from the locality of experience

2. Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective

3. Large Language Models Cannot Self-Correct Reasoning Yet 

Meta-Prompt Tuning

1. Large language models as optimizers

2. Reflexion: Language Agents with Verbal Reinforcement Learning

Process Reward

1. Let's Verify Step by Step

2. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotation

Search

1. AlphaGeometry: An Olympiad-level AI system for geometry

2. Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping

3. Mathematical discoveries from program search with large language models 

4. Self-Consistency Improves Chain of Thought Reasoning in Language Models 

5. Large Language Models as Evolutionary Optimizers

Agent Frameworks

1. ReAct

2. DSPy

3. AutoGen

Scalable Oversight

1. Measuring Progress on Scalable Oversight for Large Language Models

2. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Embodied Agent

1. Voyager: An Open-Ended Embodied Agent with Large Language Models

2. Scaling Instructable Agents Across Many Simulated Worlds

3. Eureka: Human-Level Reward Design via Coding Large Language Models 

4. GenSim: Generating Robotic Simulation Tasks via Large Language Models 

Code Generation

1. Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions

2. ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifiers

3. Scattered Forest Search: Smarter Code Space Exploration with LLM Inference

4. Is Self-Repair a Silver Bullet for Code Generation?

5. Learning Performance-Improving Code Edits

Tool-Use

1. ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings

2. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

3. Toolformer: Language Models Can Teach Themselves to Use Tools 

4. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face 

5. Gorilla: Large Language Model Connected with Massive APIs 

6. OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning 

Theorem Proving & Mathematical Reasoning

1. A Survey on Deep Learning for Theorem Proving

2. AlphaGeometry: An Olympiad-level AI system for geometry

3. Mathematical discoveries from program search with large language models 

4. In-Context Agent for Theorem Proving

5. STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving 

Test-time Scaling

1. s1: Simple test-time scaling

2. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

Auto Agent Design

1. Automated Design of Agentic Systems

2. AFlow: Automating Agentic Workflow Generation

Learning to Think

1. DeepSeek R1

2. TextGrad

3. ReST-MCTS*

4. Think before you speak: Training Language Models With Pause Tokens

5. Strategist

Agents for Science Workflows

1. An Autonomous Agent for Spatial Biology

2. Automatic Hypothesis Validation with Agentic Sequential Falsifications

3. Spatial Transcriptomics Agent

Visual Programming & Spatial Reasoning

1. Visual Programming

2. ViperGPT

3. VADAR

4. DiSciPLE

5. L3GO: Chain of 3D Thought

6. SceneCraft