Abstract

This tutorial addresses the limitations of static, machine-to-machine evaluation metrics by introducing a human-centric and cognition-driven paradigm. We propose the 3E Framework (Environment–Evaluation–Executor) to systematically assess AI systems not just by performance, but by their alignment with human reasoning, perception, and adaptability.

In the Individual Alignment part, we focus on Dynamic Visual Ability (DVA) and model human-like visual intelligence through Visual Turing Tests and eye-tracking experiments. By benchmarking AI systems against human cognitive patterns in tasks like global visual tracking and multimodal reasoning, we uncover alignment challenges and promote personalized modeling strategies.

In the Group Alignment section, we extend the evaluation to multi-agent and multi-user settings. Through benchmarks like FIOVA, we evaluate the fairness, diversity adaptation, and interpretability of AI systems in long video understanding and social computing contexts. This shift from individual to group-level evaluation ensures AI aligns not just with a single user, but with the diversity of human cognition.

We further showcase practical applications in AI for Education, where multimodal interaction data and reflective learning outcomes are jointly analyzed to evaluate AI teaching agents and adaptive learning platforms.

This tutorial provides actionable insights for researchers and practitioners aiming to build trustworthy, explainable, and human-aligned AI systems, bridging theoretical innovation with real-world deployment.

Course Description

This tutorial introduces a human-aligned and diversity-adaptive evaluation paradigm for trustworthy AI, moving beyond static benchmarks like accuracy or reward signals. We focus on bridging individual-level cognitive modeling and group-level diversity adaptation, offering a generalizable methodology across vision, education, and social computing tasks.

• In Part 1: Individual Alignment, we focus on visual intelligence as a foundation for evaluating how well AI systems match human cognition. Using Dynamic Visual Ability (DVA), Visual Turing Tests, and eye-tracking experiments, we benchmark perception, memory, and reasoning capabilities across increasingly complex tasks: from short-term tracking (STT) to long-term reasoning (GIT, MGIT). This progression models human-like visual intelligence in structured and measurable ways.

• In Part 2: Group Alignment, we shift to evaluating AI across diverse populations and complex social scenarios. While multi-annotator benchmarks like FIOVA provide a starting point for assessing alignment with diverse human perspectives in video understanding, this section expands further into social computing and education contexts, including multi-agent simulations, LLM-powered generative agents, and personality modeling. We examine how AI systems adapt to group variability, fairness requirements, and evolving collective behavior, emphasizing diversity, inclusivity, and social reasoning as central evaluation goals.

• In Conclusions, we summarize a full-spectrum evaluation strategy—from individual cognitive alignment to group-level diversity modeling—and outline future directions in capability-based, multimodal, and human-in-the-loop evaluation standardization. This serves as a foundation for building AI systems that are not only performant, but also explainable, socially aware, and ethically deployable.

The tutorial equips researchers and practitioners in XAI, computer vision, education, and social AI with actionable tools grounded in the 3E Framework (Environment–Evaluation–Executor) to design next-generation evaluation strategies for reliable, adaptable, and human-aligned AI.

 Organizers and Presenters

Schedule

14:00-15:30, 18th August, 2025. 

Room 519A, Palais des congrès, Montreal.

Anticipated Target Audience

This tutorial is designed for researchers, graduate students, and industry professionals working in artificial intelligence, particularly those interested in evaluation methodologies, computer vision, multimodal systems, and AI for Education. Participants should have a foundational understanding of machine learning principles and be familiar with commonly used evaluation metrics such as accuracy, reward scores, and precision-recall. Prior exposure to topics such as human-in-the-loop evaluation, Visual Turing Tests, or multi-agent systems is helpful but not required. The tutorial aims to provide both conceptual depth and practical guidance, making it suitable for attendees from both academic and applied research communities. 

More Information

Slides and related materials will be added here.