Neural Scaling Laws

Neural Scaling Laws

• EpochAI Scaling Laws Literature review   and  A database of papers on scaling laws

• Holistic Evaluation of Language Models (HELM)   - leaderboard

• Reproducible scaling laws for contrastive language-image learning (LAION CLIP)
  
  

• Scaling laws for single-agent reinforcement learning 

• An Information-Theoretic Analysis of Compute-Optimal Neural Scaling Laws

• Scaling Laws for Generative Mixed-Modal Language Models

• Training Trajectories of Language Models Across Scales

• Scaling Laws Beyond Backpropagation

• Broken Neural Scaling Laws

• Beyond neural scaling laws: beating power law scaling via data pruning

• Training compute-optimal large language models ("Chinchilla paper")

• What Language Model to Train if You Have One Million GPU Hours?

• Revisiting neural scaling laws in language and vision

• A Solvable Model of Neural Scaling Laws

• Transcending scaling laws with 0.1% extra compute

• Unified Scaling Laws for Routed Language Models - Scaling laws for MOEs

• Scaling laws and persistence in human brain activity

• Scaling Scaling Laws with Board Games - Scaling laws for AlphaZero on Hex

• Scaling Laws for Neural Language Models (Kaplan et al, the original famous scaling laws paper)

• Scaling Laws for Autoregressive Generative Modeling

• Scaling Laws for Transfer

• Explaining Neural Scaling Laws

• A Neural Scaling Law from the Dimension of the Data Manifold

• Scaling vision transformers 

• Deep Learning Scaling is Predictable, Empirically

• Learning Curve Theory

• On Power Laws in Deep Ensembles

• A constructive prediction of the generalization error across scales  

• Jonathan Rosenfeld's PhD thesis on  Scaling Laws for Deep Learning 

• Learning Curves: Asymptotic Values and Rate of Convergence

Scaling and Reinforcement Learning

• Human-Timescale Adaptation in an Open-Ended Task Space

• Online Decision Transformer 

• Can Wikipedia Help Offline Reinforcement Learning? 

• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

• Training language models to follow instructions with human feedback

• Offline Pre-trained Multi-Agent Decision Transformer

• Fine-Tuning Language Models from Human Preferences  

• Learning to summarize from human feedback 

• Recursively Summarizing Books with Human Feedback  

Miscelaneous

Synergy and symmetry in deep learning: Interactions between the data, model, and inference algorithm