Developers are increasingly constructing Compound AI Systems, i.e. systems that use multiple model calls and/or external components, to tackle the most challenging AI tasks. These systems can often outperform models alone, but the best way to design these systems and the components that go into them (e.g., retrievers and tools) is still an open problem.

This workshop will gather researchers working in this emerging area to investigate questions such as: 

• Designs for compound AI systems

• Efficient execution and training of compound AI systems

• Programming models for AI systems

• Optimizing AI systems end-to-end

• Planning and execution (decomposing tasks)

• Evaluation methods and metrics

• Privacy and security

• Improvements to core components like retrievers

• Methods for tool use

• Debugging and MLOps

• Scaling laws of compound AI systems

• Applications

Attending the Workshop

The inaugural workshop event will be co-located with the Databricks Data + AI Summit and hosted at the Moscone Center in San Francisco, on June 13th from 11:30am - 6pm PDT.

Select Invited Speakers and Panelists (others pending)

Organizers

Important Dates and Deadlines

• Extended abstract submission deadline: May 17th 2024

• Extended abstract acceptance notification date: May 20th 2024

• Extended abstract submission site: https://forms.gle/dMTmjhc2eAYvaRF77

• Q&A: compoundaisystems@gmail.com

• Workshop date: June 13th 2024

• Workshop Time: 11:30am - 6pm PDT

• Location: Moscone Center South in San Francisco, 747 Howard St, San Francisco, CA 94103, Room 303

Schedule (June 13th 2024, 11:30am-6:00pm PDT)

• 11:30 - 1:00 Lunch

• 1:00 - 1:15 Introduction / Kickoff: Jared Quincy Davis (Foundry / Stanford), on behalf of organizers

• 1:15 - 1:45 Invited Talk: Thang Luong (Google DeepMind)

• 1:45 - 2:15 Invited Talk: Monica Lam (Stanford)

• 2:15 - 2:45 Lightning talks [Poster Intros]

• 2:45 - 3:45 Poster Session [Coffee Break]

• 3:45 - 4:15 Invited Talk: Bryan McCann (You.com)

• 4:15 - 4:45 Invited Talk: Ken Goldberg (UC Berkeley)

• 4:45 - 5:30 Industry + Research Panel:  
  with Noam Brown (OpenAI), Yejin Choi (UW), Hannaneh Hajishirzi. (UW), Maithra Raghu (Samaya AI), Matt Bell (Anthropic), Harrison Chase (Langchain), Yu Su (OSU), and moderated by Matei Zaharia (UC Berkeley / Databricks).

• 5:30 - 6:00 Closing and Networking 

Call for Extended Abstracts 

We are accepting submissions for extended abstracts (up to 2 pages). Authors with accepted submissions will be invited to present a poster and lightning talk at the workshop.

Abstract submission deadline: May 17th 2024

Submit your abstract: https://forms.gle/dMTmjhc2eAYvaRF77 (you must be signed into an email account to make a submission)

Acceptance notification date: May 20th 2024

Accepted Posters 

• Advancing AI Capabilities through Recursive Hierarchical Task Planning and Dynamic Reasoning in Compound AI Systems. Zooey Nguyen, Vinh Luong, Shruti Raghavan, Quynh Le, Christopher Nguyen.

• Virtual Machinations: Using Large Language Models as

Neural Computers. Erik Meijer.

• ROUTERBENCH: A Benchmark for Multi-LLM Routing Systems in

Compound AI. Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, Shriyash Kaustubh Upadhyay.

• APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets. Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Zhiwei Liu, Weiran Yao, Yihao Feng, Rithesh Murthy, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, Caiming Xiong.
  
  

• Structured Data as a Key Element of AI Systems: A Test Case on Table Understanding. Barak Lenz, Raz Alon, Noam Rozen, Yonatan Belinkov, Kevin Leyton-Brown, Yoav Shoham.
  
  

• SpecTool: Characterizing Errors in Tool-Use LLMs. Shirley Kokane, Ming Zhu, Jianguo Zhang, Thai Hoang, Zuxin Liu, Tian Lan, Juntao Tan, Rithesh Murthy, Liangwei Yang, Zhiwei Liu, Weiran Yao Yihao Feng, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, Caiming Xiong.
  
  

• A Blueprint Architecture of Compound AI Systems for Enterprise. Eser Kandogan, Sajjadur Rahman, Nikita Bhutani, Dan Zhang, Rafael Li Chen, Kushan Mitra, Sairam Gurajada, Pouya Pezeshkpour, Hayate Iso, Yanlin Feng, Hannah Kim, Chen Shen, Jin Wang, Estevam Hruschka.
  
  

• LLM-Modulo Frameworks as Compound AI Architectures for Robust Planning. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Siddhant Bhambri, Kaya Stechly, Lucas Saldyt, Atharva Gundawar.
  
  

• Liberal Entity Matching as a Compound AI Toolchain. Silvery Fu, David Wang, Wen Zhang, Kathleen Ge.
  
  

• ASG: Controlling LLM Agent with Adaptive State Graph. Zhiwei Liu, Weiran Yao, Jianguo Zhang, Rithesh Murthy, Liangwei Yang, Tian Lan, Zuxin Liu, Ming Zhu, Shirley Kokane, Thai Hoang, Juan Carlos, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong.
  
  

• Composition of Experts: A Compound AI Systems Approach to build Large Language Models. Kaizhao Liang, Ravi Raju1, Swayambhoo Jain, Jonathan Li, Urmish Thakkar, Anand Sampat, Raghu Prabhakar, and Sumti Jairath.
  
  

• HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, Yu Su.
  
  

• Reasoning Capacity in Multi-Agent Systems. Pouya Pezeshkpour, Eser Kandogan, Nikita Bhutani, Sajjadur Rahman, Tom Mitchell, Estevam Hruschka.
  
  

• The Sky is the Limit: Cloud-Assisted Autonomous Driving via Service Tiers. Alexander Krentsel, Peter Schafhalter, Joseph E Gonzalez, Sylvia Ratnasamy, Scott Shenker, Ion Stoica.
  
  

• DSPy Guardrails: Building Safe LLM Applications via Self-Refining Language Model Pipelines. Boxi Yu, Pinjia He.
  
  

• JungleGPT: Designing and Optimizing Compound AI Systems for E-Commerce. Sherry Ruan, Tian Zhao.
  
  

• Meadow: LLM Agents for Data Tasks. Laurel Orr, Ines Chami
  
  

• Automating Data Discovery and Transformation: A Multi-Agent Approach. Doris Xin, Moustafa AbdelBaky.
  
  

• TextGrad: Automatic Differentiation with Text Feedback. Mert Yuksekgonul*, Federico Bianchi*, Joseph Boen*, Sheng Liu*, Zhi Huang*, Carlos Guestrin, James Zou.
  
  

• FastAgent: Enhanced Scheduling Strategies for Efficient Tool-Integrated Large Language Model Serving. Shiji Xin*, Qianru Lao*, Yueying Li*, Rana Shahout, Jiezhi Yang, Edward Suh, Christina Delimitrou, Minlan Yu.
  
  

• Compound Schema Registry. Silvery Fu, Xuewei Chen.
  
  

• From Serving Models to Serving Agents: The Missing Pieces for Supporting Agentic Workloads. Charles Packer, Sarah Wooders, Simon Mo, Kevin Lin, Ion Stoica, Joseph Gonzalez.
  
  

• SGLang: Efficient Execution of Structured Language Model Programs. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng.
  
  

• Caravan: Practical Online Learning of In-Network ML Models with Labeling Agents. Qizheng Zhang, Ali Imran, Enkeleda Bardhi, Tushar Swamy, Nathan Zhang, Muhammad Shahbaz, Kunle Olukotun.
  
  

• RoostGPT: Compound AI System as Software Testing Copilot. Rishi Yadav, Sudhir Jangir.

• Bootstrapping, Query Generation, and Continuous Improvement Within an Analytics Insight Engine. Karime Maamari, Amine Mhedhbi

• Taking Generative Feedback Loops to Production. Connor Shorten, Charles Pierse, Tommy Smith, John Trengrove, Bob van Luijt.
  
  

• Designing Media Analytics Platform for Scale. Mehul Smriti Raje. 

• A Declarative System for Optimizing AI Workloads, Chunwei Liu*, Matthew Russo*, Michael Cafarella, Lei Cao†, Peter Baille Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, Gerardo Vitagliano

Free Registration Information (**Update**: Workshop FULL; waitlist here)

1. Go to the Data + AI Summit Registration page and click Register or Log in

2. Make an account if you do not have one already and login

3. Fill in all Registration Details

4. On the last page, select Keynote + Expo Only - Meetup (Access to the Expo Hall and Keynote only) option under Conference Registration. It should be FREE (cost $0.00). If you do not see this option, make sure you clicked the link above for registration or email us for clarification. Note that this option only gets you into the workshop which is free, but not the main Data AI Summit

5. Under Session Selection, select Compound AI Systems Workshop (Thursday, June 13, 11:30 AM)

6. Press Submit Order

7. You should now see a Confirmed registration summary page

Update: We will notify people of acceptance from the waitlist by EOD Monday, June 10th.