Readings & Resources

Related Courses

Stanford CS 324

Waterloo CS 886

Benchmarks

Knowledge

1. MMLU: https://github.com/hendrycks/test?tab=readme-ov-file

2. SummEdits: https://github.com/salesforce/factualNLG

Reasoning Capability

1. Agent Bench: https://github.com/THUDM/AgentBench

2. SWE Bench: https://github.com/princeton-nlp/SWE-bench

3. LeanDojo: https://leandojo.org/

4. MATH: https://github.com/hendrycks/math/

5. HumanEval: https://github.com/openai/human-eval

6. Chain-of-Thought Hub: https://github.com/FranxYao/chain-of-thought-hub

7. WebArena: https://arxiv.org/abs/2307.13854 

Alignment

1. SafetyBench: https://github.com/thu-coai/SafetyBench

Benchmark Surveys

1. MetaTool: https://arxiv.org/pdf/2310.03128.pdf

2. LLM-eval-survey: https://github.com/MLGroupJLU/LLM-eval-survey

References

(To be updated.)
(Note: some papers belong in more than one category)

LLM Capabilities & Eval

1. Are Emergent Abilities of Large Language Models a Mirage?

2. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Reasoning (Prompt)

1. Self-Discover: Large Language Models Self-Compose Reasoning Structures

2. Take a step back: evoking reasoning via abstraction in large language models

More Reasoning

1. Measuring and Improving the Faithfulness of Model-Generated Reasoning

2. Chain-of-Thought Reasoning Without Prompting

Understanding Reasoning

1. Why think step by step? Reasoning emerges from the locality of experience

2. Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective

3. Large Language Models Cannot Self-Correct Reasoning Yet 

Meta-Prompt Tuning

1. Large language models as optimizers

2. Reflexion: Language Agents with Verbal Reinforcement Learning

Process Reward

1. Let's Verify Step by Step

2. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotation

Search

1. AlphaGeometry: An Olympiad-level AI system for geometry

2. Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping

3. Mathematical discoveries from program search with large language models 

Scalable Oversight

1. Measuring Progress on Scalable Oversight for Large Language Models

2. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Embodied Agent

1. Voyager: An Open-Ended Embodied Agent with Large Language Models

2. Scaling Instructable Agents Across Many Simulated Worlds

3. Eureka: Human-Level Reward Design via Coding Large Language Models 

4. GenSim: Generating Robotic Simulation Tasks via Large Language Models 

Code Generation

1. Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions

2. ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifiers

Tool-Use

1. ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings

2. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

3. Toolformer: Language Models Can Teach Themselves to Use Tools 

4. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face 

5. Gorilla: Large Language Model Connected with Massive APIs 

Theorem Proving & Mathematical Reasoning

1. A Survey on Deep Learning for Theorem Proving

2. AlphaGeometry: An Olympiad-level AI system for geometry

3. Mathematical discoveries from program search with large language models