Understanding Scheduling in Relational LLM Serving

Project Description:  

Large language models (LLMs) are increasingly used as operators within data-processing pipelines, for example to filter, transform, or join records based on natural-language criteria. Our recent work, Kalypso, improves the efficiency of these workloads by pipelining LLM operators and adaptively scheduling their execution based on GPU memory availability.

In this project, students will experimentally study how characteristics of a query affect the performance of different scheduling strategies. In particular, they will focus on two properties: filter selectivity, which determines how many input records proceed to downstream operators, and join fanout, which determines how many downstream tasks each record generates. These properties affect both the amount of parallel work available and the pressure placed on the GPU's KV cache.

The team will build controlled synthetic workloads in which selectivity and fanout can be varied independently. They will compare Kalypso's adaptive scheduler with simpler scheduling strategies that use fixed memory allocations between stages. The goal is to test the hypothesis that the best fixed allocation changes substantially with workload characteristics, while adaptive scheduling performs well across a broad range of conditions.

Learning Objectives:  

Students should be able to:

- Analyze how workload characteristics such as filter selectivity and join fanout affect system performance.

- Create controlled experiments that isolate specific system parameters and test clear hypotheses.

- Use GPU-based LLM serving infrastructure and collect relevant performance metrics such as latency, throughput, GPU utilization, and memory usage.

- Explore alternative scheduling strategies using quantitative experimental results.

- Analyze experimental data to identify performance trends and explain their underlying causes.

- Present systems research results clearly through technical writing, plots, and oral presentations.

Skills needed:  

Students will not be expected to have prior experience with LLM systems or GPU programming. The project will provide training in the following areas:

- Python programming and scripting for implementing experiments and automating data collection.

- Basic Linux and command-line tools for running experiments on remote GPU servers.

- Introductory concepts in LLMs and LLM serving, including inference, batching, and KV caching.

- Basic concepts in database query processing, particularly pipelines, filters, joins, and selectivity.

- Performance measurement and analysis, including latency, throughput, GPU utilization, and memory usage.

Students should ideally enter the project with basic programming experience from introductory CS courses. The more specialized systems and LLM-serving knowledge will be developed during the project.