Trusting AI by Testing and Rating Third Party Offerings

For any technology to succeed in society, people have to trust it for safety and performance. This is as much applicable for planes, medicines and children’s toys as Artificial Intelligence (AI) offerings (services and applications). As new AI applications have been deployed for mass scale use, there has been increase in trust issues with AI, regarding properties such as fairness and explainability.

The aim of the tutorial is to help newcomers to AI, as well experienced researchers, understand issues related to trust, summarize methods to tackle them, be familiar with available open source tools and give pointers to take up research in this area. The tutorial will consider a wide variety of popular types of Natural Language Processing-based AI services for illustration, such as machine translation, entity extraction, object identification and conversation agents (chatbots).

There are three points-of-view to look at a technology: producer (i.e., developer), consumer (e.g., lay user) and third party. This tutorial will uniquely focus on an independent third party view of AI technologies made available by producers and available for use by consumers, who can be end users or developers who build complex AI services.