Keynote: Knowledge Base Question Answering: Case Studies in Transfer Learning and Unanswerability

Most existing KBQA models only study heavily supervised in-domain settings where all questions are answerable. This severely limits the real-world applicability of these systems. In this talk we study both these problems: (1) transfer learning where only a small in-domain training data is available along with larger out-of-domain training set, and (2) robust KBQA, where questions may not be answerable given the KB. In addition to providing new datasets, we build a series of models using both small and large language models for both these tasks. In our final result, we are able to devise GPT4-based workflows for datasets that include unanswerable questions, which can be developed with very little in-domain training data.

Keynote: Challenges and Opportunities in Developing Conversational AI for Fintech

Building conversational AI assistants for finance with Large Language Models (LLMs) brings both opportunities and challenges. This talk delves into adapting general-purpose LLMs to the unique needs of financial chatbots, focusing on making results reliable and increasing chatbot utility. We'll explore strategies for detecting and mitigating hallucinations, measuring and communicating uncertainty, and striking the balance between helpfulness and restraint. Additionally, we'll examine how AI can offer personalized financial advice by leveraging user profiles, aligning with industry best practices, and managing uncertainty in complex interactions. We'll also discuss data challenges and suggest avenues for future research in this exciting field.

Keynote: Factual and Faithful Generation with Knowledge Retrieval

Despite demonstrating remarkable flexibility and capability, large language models (LLMs) frequently produce responses that are inconsistent or factually inaccurate. Recently, retrieval-augmented generation (RAG) has emerged as a popular mitigating strategy by equipping LLMs with relevant knowledge, but it still falls short of providing a comprehensive solution. In this presentation, I will discuss our recent endeavors to improve the factual accuracy and faithfulness of LLMs. This includes an overview of how we can automatically assess the factuality of LLM responses and encourage LLMs to rely more heavily on explicit knowledge when generating responses. Additionally, I will introduce a novel factuality-aware alignment method, comprising factuality-aware SFT and factuality-aware RL through direct preference optimization. Our approach guides LLMs to produce more factual responses while maintaining their ability to follow instructions. Finally, I will conclude by discussing key open problems and outlining directions for addressing them in the near future.

Keynote: Mixture-of-Agents Enhances Large Language Model Capabilities

Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction. Toward this goal, we propose a new approach that leverages the collective strengths of multiple LLMs through a Mixture-of-Agents (MoA) methodology. In our approach, we construct a layered MoA architecture wherein each layer comprises multiple LLM agents. Each agent takes all the outputs from agents in the previous layer as auxiliary information in generating its response. MoA models achieves state-of-art performance on AlpacaEval 2.0, MT-Bench and FLASK, surpassing GPT-4 Omni. For example, our MoA using only open-source LLMs is the leader of AlpacaEval 2.0 by a substantial gap, achieving a score of 65.1% compared to 57.5% by GPT-4 Omni.