The 1st international workshop on AI Evaluation Beyond Metrics (EBeM) will be held in Vienna, Austria (July 23-25, 2022).

Cutting edge AI and ML systems are able to solve a variety of problems that were not solvable a few years ago, such as machine translation and medical image analysis. With these AI systems starting to be deployed across important and consequential contexts, robust evaluation of their capabilities and limitations is critical. More generally, traditional approaches to evaluation lack the necessary robustness to analyse the capabilities of complex AI systems. Many AI systems solve a task or excel at a particular benchmark, but then fail at other tasks or instances that putatively represent the same capability.

Therefore, the goal of this workshop is to challenge the widespread but limited approach of evaluating the performance of intelligent systems with aggregated metrics over a benchmark or distribution of tasks. We will discuss further alternative approaches that draw on ideas and recent progress in cognitive and developmental psychology, psychometrics, software testing, and other areas.

Topics (not exhaustive)

• Evaluation methods founded on cognitive, developmental or comparative psychology

• Measurement of skills, capabilities, or cognitive abilities

• Evaluation methods based on software testing or other engineering practices

• Meta-analysis or comparisons of evaluation instruments

• The role of evaluation in AI development, policy making, and modeling of social impact

• Measurements of generality or common-sense

• Capture and use of evaluation data

• Analysis of the task space and its relation to corresponding capabilities

• The role of causality in evaluation

• Topics complementary to evaluation such as documentation or auditing

• Alternative evaluation methods with added benefits

• Discussion and progress in hard to evaluate scenarios

All registration is handled by IJCAI (more info), and the actual platform for doing so is https://registration.ijcai.org.

The EBeM venue is Messe Wien Exhibition and Congress Center:

Messe Wien

Hall B, entrance Congress Center

Messeplatz 1

A-1020 Vienna

Metro stop U2 “Messe Prater”