Scaling Laws and Foundation Models

2022 AI Safety Fundamentals course at Cambridge

https://github.com/jacobhilton/deep_learning_curriculum

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

Colin Raffel:  talks

A few possibly controversial opinions about large language models at Carnegie Mellon University Language Technologies Topical Seminar, 2021.

The Sweet Lesson at SustaiNLP Workshop, 2021.

What do language models learn from language modeling? at Stanford University CS 330 Lecture, 2021.

How and why should(n't) we scale machine learning? at IBM AI Hardware Forum Keynote, 2021.

A better way to get language models to do what you ask at AKBC 2021 Unstructured and Structured Knowledge Bases Workshop and Cohere.ai, 2021.

Scaling up Models and Data at CIFAR Deep Learning and Reinforcement Learning Summer School, Nepal Winter School in AI, and Advanced Language Processing Winter School, 2021.

More on "beyond scaling"

• The Bitter Lesson 

• Scaling Hypothesis

• The scaling hypothesis: a plan for building AGI

• OpenAI's GPT-3 Language Model: A Technical Overview

• Conversations with GPT-3

• Do large language models understand us? 

• The FLOPs Calculus of Language Model Training

• GPT 3 Demo and Explanation - An AI revolution from OpenAI

• GPT-3 Language Models are Few-Shot Learners (Paper Explained) - by Yannic Kilcher

• OpenAI GPT-3 Now Open to Public [FREE]

• Neural Scaling Laws and GPT-3

• GPT-3 Demo: New AI Algorithm Changes How We Interact With Technology

• 14 Cool Apps Built on OpenAI's GPT-3 API

• AI Chat Goes Horribly Wrong (GPT-3 interview)

• AI Robin Williams 2021 (GPT-3 / GPT 3 / GPT-J-6B) artificial intelligence

• On The Opportunities and Risks of Foundation Models
  A report by Stanford's Center for Research on Foundation Models (CRFM)

• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing

• GPT-3 paper: Language Models are Few-Shot Learners 

• CLIP paper: Learning Transferable Visual Models From Natural Language Supervision  (OpenAI blog)

• DALL-E paper: Zero-Shot Text-to-Image Generation  (OpenAI blog)

• Training language models to follow instructions with human feedback

• Pre-train, prompt, and  predict: A systematic survey of prompting methods in natural  language processing

Online Decision Transformer 

Can Wikipedia Help Offline Reinforcement Learning? 

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

Training language models to follow instructions with human feedback

Offline Pre-trained Multi-Agent Decision Transformer

Fine-Tuning Language Models from Human Preferences  

Learning to summarize from human feedback 

Recursively Summarizing Books with Human Feedback  

•  Unsolved Problems in ML Safety

• Concrete Problems in AI Safety

• A General Language Assistant as a Laboratory for Alignment

• Aligning Language Models to Follow Instructions 

• Alignment of Language Agents  

• Alignment Newsletter

• The King is Naked: on the Notion of Robustness for Natural Language Processing

Derek Parfit:     

Why Anything? Why This?

 Reasons and Persons

On What Matters (vol 1, vol 2, vol 3)

• Broken Power Laws:

• astrophysics https://www.aanda.org/articles/aa/olm/2011/02/aa15581-10/aa15581-10.html  

• material science: https://tel.archives-ouvertes.fr/tel-01037944/document

• socio-economics https://www.sciencedirect.com/science/article/abs/pii/S0378437119317935 

• hydrology https://www.researchgate.net/publication/261006967_The_use_of_broken_power-laws_to_describe_the_distributions_of_daily_flow_above_the_mean_annual_flow_across_the_conterminous_US 

• A Connectomic Hypothesis for the Hominization of the Brain

• Critical integration in neural and cognitive systems: Beyond power-law scaling as the hallmark of soft assembly

• Critical Truths about Power Laws

• Scale invariance in natural and artificial collective systems: a review

• Toward an Instance Theory of Automatization 

• “It’s like there’s this universal scaling law of cortical maturation.” https://www.spectrumnews.org/news/new-studies-reveal-how-autism-might-alter-synapse-formation-pruning/    New studies reveal how autism might alter synapse formation, pruning 

More on criticality:

• Beggs, J. M. (2008). The criticality hypothesis: how local cortical networks might optimize information processing. Philos. Trans. A Math. Phys. Eng. Sci.366, 329–343. doi: 10.1098/rsta.2007.2092

• Shew, W. L., and Plenz, D. (2013). The functional benefits of criticality in the cortex. Neuroscientist19, 88–100. doi: 10.1177/1073858412445487

• Shew, W. L., Yang, H., Petermann, T., Roy, R., and Plenz, D. (2009). Neuronal avalanches imply maximum dynamic range in cortical networks at criticality. J. Neurosci.29, 15595–15600. doi: 10.1523/JNEUROSCI.3864-09.2009

• Shew, W. L., Yang, H., Yu, S., Roy, R., and Plenz, D. (2011). Information capacity and transmission are maximized in balanced cortical networks with neuronal avalanches. J. Neurosci.31, 55–63. doi: 10.1523/JNEUROSCI.4637-10.2011

• https://jalammar.github.io/illustrated-transformer/A survey on visual transformer

• https://jalammar.github.io/illustrated-transformer/

• Transformers in vision: A survey

• An attentive survey of attention models

• Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications

• A ConvNet for the 2020s 

• Attention bottlenecks for multimodal fusion

• Table Pretraining: A Survey on Model Architectures, Pretraining Objectives, and Downstream Tasks

• Data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

• Transformers in Time Series: a Survey 

• Pretrained Transformers As Universal Computation Engines

• Time-series Transformers: a Survey

  • Multivariate Time Series Forecasting with Latent Graph Inference

  • SCINet (https://arxiv.org/pdf/2106.09305.pdf)

  • DEPTS (https://openreview.net/forum?id=AJAR-JgNw)

  • S4 (https://srush.github.io/annotated-s4/)

  • ETSformer (https://arxiv.org/abs/2202.01381)

  • Pyraformer (https://openreview.net/pdf?id=0EXmFzUn5I)

  • Informer (https://arxiv.org/abs/2012.07436)

  • Reformer (https://arxiv.org/pdf/2001.04451.pdf)

  • N-HiTS (https://arxiv.org/pdf/2201.12886.pdf)

  • Autoformer (https://arxiv.org/pdf/2106.13008.pdf)

  • LogTrans (https://arxiv.org/pdf/1907.00235.pdf)

  • GLR local global ts representations (https://arxiv.org/pdf/2202.02262.pdf)

  • TACTiS (https://arxiv.org/pdf/2202.03528.pdf)

  • MQTransformer (https://arxiv.org/pdf/2009.14799.pdf)

  • ProTran (https://proceedings.neurips.cc/paper/2021/file/c68bd9055776bf38d8fc43c0ed283Paper.pdf)

  • Preformer (https://arxiv.org/pdf/2202.11356.pdf)

  • Spacetimeformer (https://arxiv.org/pdf/2109.12218.pdf)

• A comprehensive bibliography by Gwern

• Beyond neural scaling laws: beating power law scaling via data pruning

• Training compute-optimal large language models

• Scaling Laws for Neural Language Models

• Scaling Laws for Autoregressive Generative Modeling

• Scaling Laws for Transfer

• Explaining Neural Scaling Laws

• A Neural Scaling Law from the Dimension of the Data Manifold

• Scaling vision transformers 

• Deep Learning Scaling is Predictable, Empirically

• Learning Curve Theory

• On Power Laws in Deep Ensembles

• A constructive prediction of the generalization error across scales  

• Jonathan Rosenfeld's PhD thesis on  Scaling Laws for Deep Learning

• Deconstructing Distributions: A Pointwise Framework of Learning
  
  
  

• Multiple Descent: Design Your Own Generalization Curve

• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks

• Generalization bounds for deep learning

• Bias and Generalization in Deep Generative Models: An Empirical Study

• Towards Out-Of-Distribution Generalization: A Survey 

• Domain generalization: A survey

• Generalizing to Unseen Domains: A Survey on Domain Generalization 

• Towards a Theoretical Framework of Out-of-Distribution Generalization

• OoD-Bench: Benchmarking and Understanding Out-of-Distribution Generalization Datasets and Algorithms

• More at https://sites.google.com/site/irinarish/ood_generalization (a subset to be selected)

• Other possible candidates:

https://arxiv.org/abs/2109.03795

https://arxiv.org/abs/2007.01434

https://arxiv.org/abs/2102.11436

https://arxiv.org/abs/2107.12580

https://arxiv.org/abs/2108.12284?context=cs.AI

http://proceedings.mlr.press/v119/sastry20a.htmlhttp://arxiv.org/abs/2106.03721  

https://arxiv.org/abs/2108.13624  

Scaling and Continual Learning

• Don’t Stop Learning: Towards Continual Learning for the CLIP Model 

• Continual T0 

• Effect of scale on catastrophic forgetting in neural networks 

• Effects of Model and Prior Learning Scale on Catastrophic Forgetting 

• Embracing Change: Continual Learning in Deep Neural Networks 

• Towards Continual Reinforcement Learning: A Review and Perspectives 

• Book (1st book on the topic): Lifelong Machine Learning

• Continual lifelong learning with neural networks: A review

• Continual learning: A comparative study on how to defy forgetting in classification tasks

• A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning 

• Class-incremental learning: survey and performance evaluation

• Never-Ending Learning (tutorial by Tom Mitchell and Partha Talukdar, ICML 2019)

• Continual Learning with Deep Architectures (tutorial by Irina Rish and Vincenzo Lomonaco, ICML 2021)

• Continual Lifelong Learning in Natural Language Processing: A Survey   

• Drinking from a Firehose: Continual Learning with Web-scale Natural Language

• Pretrained Language Model in Continual Learning: A Comparative Study 

• Solvable Model for Inheriting the Regularization through Knowledge Distillation