Course Description

As AI systems are deployed in increasingly complex and uncertain environments, traditional evaluation metrics such as accuracy or reward are no longer sufficient to measure AI’s true capabilities. This tutorial explores the evolving landscape of trustworthy AI evaluation, moving beyond task-specific benchmarks to assess reasoning ability, adaptability, and robustness in dynamic, real-world settings.

We will begin by reviewing the historical development of AI evaluation, highlighting the limitations of static benchmarks in computer vision and reinforcement learning. Then, we introduce recent advances in capability-based, interactive, and multimodal evaluation methods—such as the Visual Turing Test, event-level analysis, and cross-modal alignment techniques. The first half of the tutorial focuses on computer vision tasks, including video reasoning, vision-language grounding, and multimodal robustness. The second half shifts to complex systems, covering long-term strategy assessment in multi-agent settings, network-based evaluation, and real-world applications such as epidemic modeling and intelligent traffic control.

By bridging theory, methods, and applications, this tutorial aims to provide a unified perspective on how to evaluate AI in the wild, equipping participants with practical tools and conceptual frameworks to support trustworthy AI development across disciplines.

Outline

The organization of the tutorial is as follows:

Part I. Introduction 

This opening section provides an overview of how AI evaluation has evolved from static, task-specific metrics (e.g., accuracy, precision, reward) to more dynamic and holistic approaches. It highlights critical limitations of traditional benchmarks—such as their inability to measure reasoning, generalization, and long-term robustness—and introduces the tutorial’s structure, which spans vision-based tasks, complex systems, and future evaluation paradigms. 

Part II. Computer Vision Evaluation Methods 

This module explores the challenges and recent advances in evaluating AI within visual tasks. It covers standard vision metrics, limitations in long video understanding, and emerging interactive and multimodal evaluation techniques. Participants will learn about new paradigms such as Visual Turing Tests and event-level analysis that better reflect AI’s spatial-temporal reasoning and multimodal understanding. 

Part III. AI Evaluation in Complex Systems 

This section delves into AI evaluation beyond perception tasks, focusing on decision-making and strategic adaptation in complex, dynamic environments. It introduces system-level evaluation frameworks in reinforcement learning and multi-agent settings, as well as network-based approaches for modeling social influence, epidemic control, and traffic systems. The goal is to assess AI’s stability, adaptability, and long-term reasoning under uncertainty. 

Part IV. Future Trends in AI Evaluation 

The final section looks forward to emerging trends in trustworthy AI evaluation. It emphasizes the importance of capability-driven metrics, interactive assessment techniques that reflect real-world dynamics, and standardized multimodal evaluation protocols. Novel theoretical tools—such as Parrondo’s Paradox and graph-based optimization—are discussed to inspire new perspectives on how to evaluate intelligent systems in open and evolving environments. 

 Organizers and Presenters

Schedule

Half-day. October 25-30, 2025. Bologna, Italy. (Detailed timetable and location will be updated soon)

Anticipated Target Audience

This tutorial is designed for PhD students, researchers, and industry professionals interested in AI evaluation. (1) PhD students will gain foundational knowledge beyond accuracy-based metrics, helping them understand key research directions in trustworthy AI assessment. (2) Experienced researchers will explore adaptive AI evaluation and dynamic performance measurement, essential for assessing AI’s long-term stability and generalization. (3) Industry professionals in autonomous driving, medical AI, and finance will learn how evaluation frameworks enhance AI reliability in real-world applications. Participants should have (1) basic knowledge of machine learning and deep learning, (2) familiarity with computer vision or complex decision-making systems, and (3) some probability background (helpful but not required).

More Information

Slides and related materials will be added here.