ICAIF'24 Tutorial on Evaluating and Rating AI Systems for Trust and Its Application to Finance

The goal of this tutorial is to provide participants with an in-depth understanding of how to evaluate and rate black-box financial AI systems for robustness and bias using statistical and causal analysis techniques. We will cover methods for assessing both white-box and black-box systems, with a particular emphasis on the latter. Attendees will learn to apply these methodologies to financial decision-making contexts. The session will feature a live demonstration of the ARC (AI Rating through Causality) tool, which exemplifies these evaluation techniques in action, offering participants hands-on experience in applying these methods to real-world scenarios.

 This tutorial explores the evaluation and rating of AI systems for trust, specifically focusing on financial applications. As AI technologies increasingly rely on correlational data, their black-box nature becomes a problem in high-stakes domains such as finance, where errors can have severe implications. We will discuss the rating method which is used to make the system behavior transparent. Building on previous research, this tutorial will particularly discuss our novel causal analysis-based approach to rate different black-box AI systems for bias and robustness. This tutorial aims to equip stakeholders with the necessary tools to verify and choose AI systems that demonstrate real-world reliability and robustness under various conditions.