Ethics, Governance, and Compliance Resources

The legal and regulatory landscape for AI and ML systems is changing rapidly. The list of resources here reflects the leading thinking from academics, regulatory agencies, and on-going projects and studies to provide the best guidance to commercial and public entities on implementing AI into their products and services.

I. General AI & Ethics Resources

Existing Company and Government Models or Recommended Best Practices

Intel, Artificial Intelligence: The Public Policy Opportunity - Intel’s public policy recommendations to fo

ster an environment conducive to AI innovation, while mitigating the unintended societal consequences.

Google, AI at Google: Our Principles - Google’s AI principles.

Microsoft, Microsoft AI Principles - Microsoft’s AI Principles.

DeepMind, Ethics and Society Principles - DeepMind’s AI Principles.

Facebook, AI at F8 2018 - An outline of Facebook’s vision for AI Development.

SIIA, Ethical Principles for Artificial Intelligence and Data Analytics - The Software & Information Industry Association’s ethical principles for AI and data analytics.

Public Voice, Universal Guidelines for Artificial Intelligence - Guidelines for Artificial Intelligence set up by The Public Voice, a coalition which was established in 1996 by the Electronic Privacy Information Center (EPIC) to promote public participation in decisions concerning the future of the Internet.

IEEE, The Ethics Certification Program for Autonomous and Intelligent Systems (ECPAIS) - A proposed certification system to create specifications marking processes that advance transparency, accountability and reduction in algorithmic bias in Autonomous and Intelligent Systems

The AI Policy Landscape - A general compendium of multiple AI commentary and resources. Original Medium post and discussion here and continuously updated Google doc version with table of contents here

Alisomar, Alisomar AI Principles - A further explanation on these principles can be found here., Responsible AI in Consumer Enterprise - Integrate.AI’s framework for businesses to use consumer data responsibly.

Google, Perspectives on Issues in AI Governance - This white paper calls for government regulation in the field of AI, suggesting specific areas to be considered.

NEC, NEC Unveils "NEC Group AI and Human Rights Principles" - NEC’s principles to prioritize privacy and human rights in relation to the development of AI.

OECD, OECD Principles on AI - A list of principles on AI established by the the Organisation for Economic Co-operation and Development.

Office of Artificial Intelligence (UK), Draft Guidelines for AI Procurement - Intended to be a working document drafted in collaboration with the World Economic Forum Centre for the Fourth Industrial Revolution

AI Global, Creating A Responsible Trust Index: A Unified Assessment to Assure the Responsible Design, Development, and Deployment of AI - A unified framework focusing on: Accountability; Explainability and Interpretability; Data Quality; Bias and Fairness; Robustness

Springer, AI & Ethics - A publication focusing on the informed debate and discussion of the ethical, regulatory, and policy implications that arise from the development of AI, 11 Short Videos About AI Ethics - A playlist of 11 short videos (most are 6-13 mins long) on ethics in machine learning.

News, Reports, and Other Media

Algorithmwatch, In the Realm of Paper Tigers - Exploring the Failings of AI Ethics Guidelines - Assessment and directory of 160 AI ethics guidelines.

AI 4 People, AI 4 People's Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations - This document outlines the risks and opportunities of AI, propose ethical principles of AI, and offers recommendations for a Good AI Society.

The Stanford AI Lab Blog, Shushman Choudhury, et al., In Favor of Developing Ethical Best Practices in AI Research - This blog post calls for AI Researchers to consider the ethics of their work and to create a system of ethical best practices.

Towards Data Science, A Gentle Introduction to the Discussion on Algorithmic Fairness - An overview of the problems that arise from algorithmic decisionmaking.

Nasdaq, How Artificial Intelligence Can Influence Governance, Risk, and Compliance - A consideration of the ways in which AI can aid governance, risk, and compliance (GRC) activities.

Medium, AI and the Future of Ethics - A basic overview of AI and discussion about the ethics related to it.

Tutorial, 21 Fairness Definitions and Their Politics - A filmed lecture discussing the various definitions of fairness as they pertain to statistical models.

Google Research, Attacking discrimination with smarter machine learning - Case study of a loan application scenario with a demo of how to confront bias

Medium, AI and the Future of Ethics - A basic overview of AI and discussion about the ethics related to it.

Algorithmic Justice League, Study finds gender and skin-type bias in commercial AI systems. Site for a collective that aims to increase awareness and report bias. Ted talk here.

CFA Institute, Artificial Intelligence: The Next Step in Corporate Governance -

University of Toronto Centre for Ethics, The Ethics of Agonistic Machine Learning

Medium, Toward Ethical, Transparent And Fair AI/Ml: A Critical Reading List

Clifford Rossi, A Risk Professional’s Survival Guide: Applied Best Practices in Risk Management

IAEE, Classical Ethics in A/IS

AI Now, Algorithmic Impact Assessments: A Practical Framework for Public Agency Accountability - A policy paper providing public agencies a practical framework to assess automated decision systems and to ensure accountability.

AI Now Institute, Algorithmic Impact Assessments: A Practical Framework For Public Agency Accountability

Brookings, The Role of Corporations in Addressing AI’s Ethical Dilemmas - Darrell West discusses 5 AI ethical dilemmas and how corporations are addressing them.

Future Advocacy, Ethical, Social and Political Challenges of Artificial Intelligence in Health, April 2018 - A review of existing literature and interviews with global experts to understand how AI is being used (or could be used) in healthcare and what challenges these uses present.

IEEE, Ethics in Action

The Ethics and Governance of AI Initiative

Future of Humanity Institute, Governance of AI Program

MIT Media Lab, Ethics and Governance of Artificial Intelligence

Berkman Klein Center, Ethics and Governance of Artificial Intelligence

The GovLab, Artificial Intelligence and Public Policy

MILA, Official Launch of the Montreal Declaration for Responsible Development of Artificial Intelligence

AI Now, AI Now 2017 Report

American Bar Association, A 'Principled' Artificial Intelligence could improve justice

American Action Forum, Primer: How to Understand and Approach AI Regulation

American Action Forum, Approaches to Regulating Technology: From Privacy to A.I.

Markkula Center for Applied Ethics, Ethics in Technology Practice

Alan Winfield, An Updated Round Up of Ethical Principles of Robots and AI - A list of numerous attempts at outlining ethical principles for Artificial Intelligence, beginning with Isaac Asimov’s 1950 laws of Robotics. (Most of these links should already be captured in our wiki for Ethics or Education.)

Markkula Center for Applied Ethics, Readings in AI Ethics - A compilation of readings on the ethics of AI.

Brookings, Algorithmic Bias Detection and Mitigation: Best Practices and Policies to Reduce Consumer Harms - A report that gives examples of, considers the harms of, offers detection methods for, and proposes solutions to mitigating the harms of algorithmic bias.

AI Now Institute, Discriminating Systems: Gender, Race, and Power in AI - A report that outlines the causes of and the issues resulting from the lack of diversity in the AI sector. It makes recommendations to help better the problems.

Benedict Evans, Notes on AI Bias - This blog post argues that while AI bias is an issue, the problem of bias is not new, and is not rooted in problems with machines, but instead with humans.

Ethical Resolve, Provides blog posts, talks, and resources for businesses concerned with implementing responsible AI and Ethics in Design.

Nature Machine Intelligence, The Global Landscape of AI Ethics Guidelines - Maps and analyzes the current corpus of AI ethics guidelines.

Partnership on AI, Report on Algorithmic Risk Assessment Tools in the U.S. Criminal Justice System - This report documents the serious shortcomings of risk assessment tools in the U.S. criminal justice system, most particularly in the context of pretrial definitions.

Access Now, Human Rights in the Age of Artificial Intelligence - Provides a jumping off point for further conversation and research in this developing space.

Access Now, Human Rights Matter in the AI Debate. Let's Make Sure AI Does Us More Good Than Harm

Access Now, Laying Down the Law on AI: Ethics Done, Now the EU Must Focus on Human Rights

AI Now Institute, Algorithmic Accountability Policy Toolkit - This toolkit is intended to provide legal and policy advocates with a basic understanding of government use of algorithms including, a breakdown of key concepts and questions that may come up when engaging with this issue, an overview of existing research, and summaries of algorithmic systems currently used in government.

Data & Society, Algorithmic Accountability: A Primer - The primer explores the trade-offs debates about algorithms and accountability across several key ethical dimensions, including fairness and bias; opacity and transparency; and lack of standards for auditing.

Data & Society, Governing Artificial Intelligence: Upholding Human Rights & Dignity - This report shows how human rights can serve as a “North Star” to guide the development and governance of artificial intelligence.

Karen Hao, This is How AI Bias Really Happens—and Why It's So Hard to Fix - Bias can creep in at many stages of the deep-learning process, and the standard practices in computer science aren’t designed to detect it.

UNICEF & Human Rights Center, Executive Summary: Artificial Intelligence and Children's Rights - This memo outlines a series of case studies to illustrate the various ways that artificial intelligence-based technologies are beginning to positively and negatively impact children’s human rights.

FAT ML, Principles for Accountable Algorithms and a Social Impact Statement for Algorithms - The goal of this document is to help developers and product managers design and implement algorithmic systems in publicly accountable ways.

Salesforce, Ethical Frameworks, Tool Kits, Principles, and Certifications—Oh My! - This post links to dozens of ethics resources from both the private and public sector.

International Resources

ICO, Draft Guidance for Consultation, Guidance on the AI Auditing Framework - advice on how to understand data protection law in relation to artificial intelligence (AI) and recommendations for organisational and technical measures to mitigate the risks AI poses to individuals.

European Commission, White Paper On Artificial Intelligence - A European Approach to Excellence and Trust - A European approach to artificial intelligence building off the European strategy for AI presented in April 2018.

European Union Agency for Fundamental Rights, Data Quality and Artificial Intelligence—Mitigating Bias and Error to Protect Fundamental Rights

Norwegian Data Protection Authority, Data Protection by Design and by Default

Berkeley Technology Law Journal, Rethinking Explainable Machines: The GDPR's 'Right to Explanation' Debate and the Rise of Algorithmic Audits in Enterprise

World Economic Forum, Top 9 Ethical Issues In Artificial Intelligence

Office of the Victorian Information Commissioner, Artificial Intelligence and Privacy

European Agency for Fundamental Rights, #Big Data: Discrimination in Data-Supported Decision Making

United Nations University, Centre for Policy Research, The Ethical Anatomy of AI

Dutch Alliance for Artificial Intelligence

Australian Institute of Company Directors,Preparing Directors For Artificial Intelligence Whirlwind

Université de Montréal, Montréal Declaration of Responsible AI

House of Lords, Select Committee on Artificial Intelligence, AI in the UK: ready, willing, and able?

European Group on Ethics in Science and New Technologies, Statement on Artificial Intelligence, Robotics and 'Autonomous' Systems

Cédric Villani, For a Meaningful Artificial Intelligence: Towards a French and European Strategy

European Commission for the Efficiency of Justice (CEPEJ), European ethical Charter on the use of Artificial Intelligence in judicial systems and their environment

European Commission High-Level Expert Group on Artificial Intelligence (AI HLEG), Draft Ethics guidelines for trustworthy AI

Automating Society, Taking Stock of Automated Decision-Making in the EU

UK Information Commissioner's Office,Automated Decision Making: the role of meaningful human reviews- This framework describes how humans can have ‘meaningful’ involvement in AI decision making.

Dr. Thilo Hagendorff, The Ethics of AI Ethics- An evaluation of the AI guidelines that have been presented. An overview of the field of AI ethics.

Government of Canada, Responsible use of artificial intelligence (AI) - Guiding principles to ensure the use of ethical AI by the government.

The White House, The Administration’s Report on the Future of Artificial Intelligence - A 2016 report focusing on the opportunities, considerations, and challenges of Artificial Intelligence.

Office of the Privacy Commissioner of Canada, Consultation on the OPC's Proposals for Ensuring Appropriate Regulation of Artificial Intelligence - proposals for regulating artificial intelligence.

II. Resources for AI Ethical Review Process

Future of Privacy Forum, Conference Proceedings - Beyond IRBs Designing Ethical Review Processes for Big Data Research - 2015 conference proceedings aimed at identifying processes and commonly accepted ethical principles for data research in academia, government and industry.

Northeastern University Ethics Institute, Building Data and AI Ethics Committees - Describes components of a committee-based approach to data and AI ethics, while identifying questions for an organization to consider when developing ethics and oversight committees.

Council of Europe, The Council of Europe Established an Ad Hoc Committee on Artificial Intelligence - CAHAI - Committee to examine the feasibility and potential elements of a legal framework for the development of artificial intelligence.

Berkman Klein Center, Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-Based Approaches to Principles for AI- Provides a comparison between thirty-six prominent AI principles documents side-by-side.

BNH.AI, Sample AI Incident Response Checklist - A checklist for 7 Phases of AI incident response including: preparation; identification; containment; eradication; and recovery. Also provides additional compliance resources.

Dallas Card & Noah A. Smith, On Consequentialism and Fairness - A consequentialist critique of common definitions of fairness within machine learning, as well as a machine learning perceptive on consequentialism; concluding with a broader discussion of how issues of learning and randomization have important implications for the ethics of automated decision making systems.