From customer support agents to coding copilots, agentic AI systems must make decisions under uncertainty while relying on noisy, delayed, and expensive feedback, precisely the settings for which decades of research in bandits and reinforcement learning (RL) have developed principled methods. Yet many of these decisions are still driven by ad hoc heuristics rather than principled methods. This tutorial presents a unified view of agentic AI through the lens of sequential decision-making. We show that five core challenges, Model Selection, Self-Improvement, Adaptation and Personalization, Orchestration, and Alignment with Human Feedback, share a common formal structure. For each challenge, we provide an explicit bandit or RL formulation, together with the corresponding algorithmic toolbox and theoretical foundations. By bridging bandit and RL theory with modern agentic AI, our goal is to promote principled, reliable, and scalable solutions for the next generation of intelligent systems.

• Part-I: Welcome and Gentle Introduction to Key Techniques (45m)
  - Motivation (5m)
  - Setting & Primer (12m),
  - Model Selection (14m),
  - Self-Improvement (14m).
  [Speaker: Aadirupa Saha]  

• Part-II: Main Technical Content (45m)  
  - Adaptation and Personalization (14m)
  - Orchestration & Multi-Agent Coordination (14m)
  - Alignment with Human Feedback (12m)
  - Conclusion & Open Problems (5m)
  [Speaker: Arun Verma]

• Part-III: Hands-On Demo (20 mins)
  [Speaker: Djallel Bouneffouf]

• Part-IV: Questions & Answers (10 mins)

• Part-V: Panel: Principled vs. Heuristic Agentic AI: Can Principled Approaches Outperform Heuristics for Reliable AI Agents? (30 mins)
  [Panel: Akshay Krishnamurthy, Branislav Kveton, John Langford, Lihong Li, Paul Liang, Armando Solar-Lezama]
  [Moderator: Aadirupa Saha]
  
  (*all times are in the local conference timezone)

Motivation: Sequential Decision-Making in Agentic AI

• Everyday agents: customer support, travel planning, coding copilots

• Why heuristics fall short: noisy, delayed, expensive feedback

• The LLM–Bandit/RL convergence

Formalism Primer

• Multi-armed, Contextual, and Dueling Bandits

• MDPs and POMDPs

• Regret, Sample Complexity, and Safe Exploration

Tool and Model Selection

• Routing as a Contextual Bandit (quality–cost trade-off)

• Bandits with Knapsacks under Budget Constraints

• Dueling Bandits under Pairwise Quality Judgments

• Systems: RouteLLM, FrugalGPT

Self-Improvement

• Best-Arm Identification over Prompts and Configurations

• Label-Free Dueling Bandits with LLM Judges

• Prompt Revision as an MDP

• Systems: OPRO, ProTeGi, TRIPLE

Adaptation and Personalization

• Contextual Bandits over User/Session Features

• Dueling Bandits under Relative Preferences

• Off-Policy Evaluation for Safe Personalization

• The Non-Myopic MDP View of Evolving User Models

Orchestration and Multi-Agent Coordination

• Step-Level Contextual Bandits

• MDPs and POMDPs under Sparse, Terminal Rewards

• Credit Assignment Across Long Horizons

• MCTS/UCT, Policy Gradient, and Actor-Critic

• Systems: AFlow, ToolOrchestra, MASPOB

Human Preference Feedback and Alignment

• Contextual Dueling Bandits under Bradley–Terry Models

• Conservative, Constrained, and Nash Variants

• KL-Regularized RLHF with PPO

• DPO as Reparameterized Supervised Learning

• Nash Preference Games and Multiplayer Optimization

Hands-on Demos & Open Problems

• Model Routing as a Contextual Bandit

• Prompt Optimization as Best-Arm Identification

• RLHF/DPO under Bradley–Terry-Luce (BTL) Preferences

• Open Problems: Safe Exploration, Credit Assignment, Personalization at Scale

Panel Discussion: Principled vs. Heuristic Agentic AI: Can Principled Approaches Outperform Heuristics for Reliable AI Agents?

The tutorial is meant to be accessible to the entire machine learning community, and especially useful for bandits and reinforcement learning researchers.

Prerequisites: A basic knowledge of probability theory, and linear algebra should be enough. Familiarity with standard concentration inequalities, state of the art multi-armed bandits (MAB) algorithms would be helpful (only to understand the algorithm technicalities), but not necessary and as mentioned, we will cover the basics of classical MAB techniques in the beginning of the talk. The tutorial will be self-contained with all the basic definitions.

Most of the target audiences are likely to be Machine Learning oriented, cutting across grad students, postdocs, or faculties. Overall, any first year grad student is expected to be comfortable. The tutorial intends to provide enough exposure to the audience to built a basic understanding of bandit-problems, the need of its preference counterpart, existing results, and exciting scopes of open challenges.

• Surveys and Foundational Reading

  • LLM and Bandit/RL connections: A Component-Based Survey of Interactions between LLMs and Multi-Armed Bandits

  • Dueling and preference-based learning: Advancements in Dueling Bandits, Preference-Based Online Learning with Dueling Bandits, A Survey of Preference-Based RL Methods

  • Contextual bandits: LinUCB, generalized linear contextual bandits, improved linear stochastic bandits

  • Thompson sampling (Agrawal and Goyal, 2012) and bandits with knapsacks (Badanidiyuru et al., 2018)

  • Best-arm identification: Almost Optimal Exploration, lil' UCB

  • RLHF and alignment foundations: Deep RL from Human Preferences, InstructGPT, DPO, Nash Learning from Human Feedback

  • Tree search as bandit exploration: MCTS, UCT, AlphaGo

• Agentic Systems and Benchmarks

  • Routers: RouteLLM, FrugalGPT, PILOT, Hybrid LLM, AutoMix

  • Prompt optimizers: APE, OPRO, ProTeGi, TRIPLE, INSTINCT, APOHF

  • Orchestration: AFlow, ToolOrchestra, MASPOB

  • Common Benchmarks: τ-bench, TravelPlanner, SWE-bench, WebArena, AgentBench, GAIA

• Adaptation and Personalization

  • Learning user models online: FSPO, T-POP, contextual-bandit user profiles for RAG

  • Eliciting preferences efficiently: Batch Active Preference-Based Learning, Dueling Bandits with Adversarial Sleeping

  • Game-theoretic alignment: Multiplayer Nash Preference Optimization

• Related Tutorials

  • Do You Prefer Learning with Preferences? NeurIPS 2023

  • Cross-Disciplinary Insights into Alignment in Humans and Machines NeurIPS 2024

  • Generating Programmatic Solutions:  Algorithms and Applications of Programmatic Reinforcement Learning and Code Generation NeurIPS 2024

  • A Tutorial on Multi-Armed Bandit Applications for Large Language Models KDD 2024

  • Bandits, LLMs, and Agentic AI AAAI 2026