[Tutorial]
Inference optimization for foundational models
on AI accelerators

Sunday, August 25, from 2:00pm to 5:00pm

Venue: Centre de Convencions Internacional de Barcelona

Room: 129-130

Tutorial Abstract

Powerful foundation models, including large language models (LLMs), with Transformer architectures have ushered in a new era of Generative AI across various industries. Industry and research community have witnessed a large number of new applications, based on those foundation models. Such applications include question and answer, customer services, image and video generation, and code completions, among others. However, as the number of model parameters reaches to hundreds of billions, their deployment incurs prohibitive inference costs and high latency in real-world scenarios. As a result, the demand for cost-effective and fast inference using AI accelerators is ever more higher.

To this end, our tutorial offers a comprehensive discussion on complementary inference optimization techniques using AI accelerators. Beginning with an overview of basic Transformer architectures and deep learning system frameworks, we deep dive into system optimization techniques for fast and memory-efficient attention computations and discuss how they can be implemented efficiently on AI accelerators. Next, we describe architectural elements that are key for fast transformer inference. Finally, we examine various model compression and fast decoding strategies in the same context.

Slides and Survey paper

See logistics/materials page

Speaker's Bio

Youngsuk Park is a senior applied scientist at AWS AI Research and Education, currently leading efficient training and inference research for foundation models on AI accelerators. He published 25+ papers on the topics of optimization, RL, LLM, stable diffusion, time series at top venues including ICLR, ICML, and AISTATS. In addition, he delivered and hosted multiple tutorials and workshops on LLM training and times series modelling. Before joining AWS, he obtained PhD in Electrical Engineering from Stanford University.

Kailash Budhathoki is a senior applied scientist at AWS AI Research and Education, where he is currently working on optimizing foundation models for inference. Prior to that, he worked on algorithmic fairness, and causality. He graduated with a PhD in Computer Science from the Max Planck Institute for Informatics, Germany.

Yida Wang is a principal scientist in the AWS AI Research and Education. His research interest is in systems, high-performance computing, and big data analytics. He works on deep learning systems, with a focus on compiling and optimizing deep learning models for efficient training and inference, especially the large-scale foundation models. The mission is to bridge the high-level models from various frameworks and low-level hardware platforms including CPUs, GPUs, and AI accelerators, so that different models can execute in high-performance on different devices.

Jonas Kübler is an applied scientist at AWS AI Research and Education, currently optimizing the inference performance of LLMs. His previous work on statistical testing and quantum machine learning was published at conferences and journals including NeurIPS, ICML, AISTATS and Nature Communications. He did his PhD at the Max Planck Institute for Intelligent Systems.