Scaling workshop series are organized by the CERC in Autonomous AI Lab led by Irina Rish at the Universite de Montreal and Mila - Quebec AI Institute.
The goal of these events is to provide a forum for discussing recent advances in foundation models - large-scale neural networks, pretrained in an unsupervised way on large and diverse datasets - and neural scaling laws for predicting models' behaviors at scale.
This event aims to provide:
An overview of the rapidly growing field of large-scale AI, foundation models, scaling laws and emergent behaviors
Tutorials on how to build such models in multi-GPU settings (see some background here on HPC and Distributed Model Training)
An overview of ongoing open-source foundation models projects at CERC-AAI lab and AGI Collective discord community, based on continual pretraining, multimodal model training an alignment, as well as time-series research. Thanks to the OLCF compute on Summit and Frontier supercomputers, so far, these efforts resulted into several open-source model releases, including:
six LLMs of 7B to 10B parameters (including 7B Red-Pajama-INCITE released May 2023, a CL-FoMo suite of fours 9.6B LLMs released in December 2023, as well as Hi-NOLIN - the first English/Hindi LLM, released in OCtober 2023);
Robin suite of varying-size vision-language models (VLMs) released in December 2023; ongoing efforts on multimodal evaluation and alignment
Time-series foundation model (Lag-Llama) released in February 2024, and more!