Ethics, Governance, and Compliance Resources
The legal and regulatory landscape for AI and ML systems is changing rapidly. The list of resources here reflects the leading thinking from academics, regulatory agencies, and on-going projects and studies to provide the best guidance to commercial and public entities on implementing AI into their products and services.
I. General AI & Ethics Resources
Existing Company and Government Models or Recommended Best Practices
Intel, Artificial Intelligence: The Public Policy Opportunity - Intel’s public policy recommendations to fo
ster an environment conducive to AI innovation, while mitigating the unintended societal consequences.
Google, AI at Google: Our Principles - Google’s AI principles.
Microsoft, Microsoft AI Principles - Microsoft’s AI Principles.
DeepMind, Ethics and Society Principles - DeepMind’s AI Principles.
Facebook, AI at F8 2018 - An outline of Facebook’s vision for AI Development.
SIIA, Ethical Principles for Artificial Intelligence and Data Analytics - The Software & Information Industry Association’s ethical principles for AI and data analytics.
Public Voice, Universal Guidelines for Artificial Intelligence - Guidelines for Artificial Intelligence set up by The Public Voice, a coalition which was established in 1996 by the Electronic Privacy Information Center (EPIC) to promote public participation in decisions concerning the future of the Internet.
IEEE, The Ethics Certification Program for Autonomous and Intelligent Systems (ECPAIS) - A proposed certification system to create specifications marking processes that advance transparency, accountability and reduction in algorithmic bias in Autonomous and Intelligent Systems
The AI Policy Landscape - A general compendium of multiple AI commentary and resources. Original Medium post and discussion here and continuously updated Google doc version with table of contents here
Integrate.ai, Responsible AI in Consumer Enterprise - Integrate.AI’s framework for businesses to use consumer data responsibly.
Google, Perspectives on Issues in AI Governance - This white paper calls for government regulation in the field of AI, suggesting specific areas to be considered.
NEC, NEC Unveils "NEC Group AI and Human Rights Principles" - NEC’s principles to prioritize privacy and human rights in relation to the development of AI.
OECD, OECD Principles on AI - A list of principles on AI established by the the Organisation for Economic Co-operation and Development.
Office of Artificial Intelligence (UK), Draft Guidelines for AI Procurement - Intended to be a working document drafted in collaboration with the World Economic Forum Centre for the Fourth Industrial Revolution
AI Global, Creating A Responsible Trust Index: A Unified Assessment to Assure the Responsible Design, Development, and Deployment of AI - A unified framework focusing on: Accountability; Explainability and Interpretability; Data Quality; Bias and Fairness; Robustness
Springer, AI & Ethics - A publication focusing on the informed debate and discussion of the ethical, regulatory, and policy implications that arise from the development of AI
fast.ai, 11 Short Videos About AI Ethics - A playlist of 11 short videos (most are 6-13 mins long) on ethics in machine learning.
News, Reports, and Other Media
Algorithmwatch, In the Realm of Paper Tigers - Exploring the Failings of AI Ethics Guidelines - Assessment and directory of 160 AI ethics guidelines.
AI 4 People, AI 4 People's Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations - This document outlines the risks and opportunities of AI, propose ethical principles of AI, and offers recommendations for a Good AI Society.
The Stanford AI Lab Blog, Shushman Choudhury, et al., In Favor of Developing Ethical Best Practices in AI Research - This blog post calls for AI Researchers to consider the ethics of their work and to create a system of ethical best practices.
Towards Data Science, A Gentle Introduction to the Discussion on Algorithmic Fairness - An overview of the problems that arise from algorithmic decisionmaking.
Nasdaq, How Artificial Intelligence Can Influence Governance, Risk, and Compliance - A consideration of the ways in which AI can aid governance, risk, and compliance (GRC) activities.
Medium, AI and the Future of Ethics - A basic overview of AI and discussion about the ethics related to it.
Tutorial, 21 Fairness Definitions and Their Politics - A filmed lecture discussing the various definitions of fairness as they pertain to statistical models.
Google Research, Attacking discrimination with smarter machine learning - Case study of a loan application scenario with a demo of how to confront bias
Medium, AI and the Future of Ethics - A basic overview of AI and discussion about the ethics related to it.
Algorithmic Justice League, Study finds gender and skin-type bias in commercial AI systems. Site for a collective that aims to increase awareness and report bias. Ted talk here.
CFA Institute, Artificial Intelligence: The Next Step in Corporate Governance -
University of Toronto Centre for Ethics, The Ethics of Agonistic Machine Learning
IAEE, Classical Ethics in A/IS
AI Now, Algorithmic Impact Assessments: A Practical Framework for Public Agency Accountability - A policy paper providing public agencies a practical framework to assess automated decision systems and to ensure accountability.
Brookings, The Role of Corporations in Addressing AI’s Ethical Dilemmas - Darrell West discusses 5 AI ethical dilemmas and how corporations are addressing them.
Future Advocacy, Ethical, Social and Political Challenges of Artificial Intelligence in Health, April 2018 - A review of existing literature and interviews with global experts to understand how AI is being used (or could be used) in healthcare and what challenges these uses present.
IEEE, Ethics in Action
Future of Humanity Institute, Governance of AI Program
MIT Media Lab, Ethics and Governance of Artificial Intelligence
Berkman Klein Center, Ethics and Governance of Artificial Intelligence
The GovLab, Artificial Intelligence and Public Policy
AI Now, AI Now 2017 Report
American Bar Association, A 'Principled' Artificial Intelligence could improve justice
American Action Forum, Primer: How to Understand and Approach AI Regulation
American Action Forum, Approaches to Regulating Technology: From Privacy to A.I.
Markkula Center for Applied Ethics, Ethics in Technology Practice
Alan Winfield, An Updated Round Up of Ethical Principles of Robots and AI - A list of numerous attempts at outlining ethical principles for Artificial Intelligence, beginning with Isaac Asimov’s 1950 laws of Robotics. (Most of these links should already be captured in our wiki for Ethics or Education.)
Markkula Center for Applied Ethics, Readings in AI Ethics - A compilation of readings on the ethics of AI.
Brookings, Algorithmic Bias Detection and Mitigation: Best Practices and Policies to Reduce Consumer Harms - A report that gives examples of, considers the harms of, offers detection methods for, and proposes solutions to mitigating the harms of algorithmic bias.
AI Now Institute, Discriminating Systems: Gender, Race, and Power in AI - A report that outlines the causes of and the issues resulting from the lack of diversity in the AI sector. It makes recommendations to help better the problems.
Benedict Evans, Notes on AI Bias - This blog post argues that while AI bias is an issue, the problem of bias is not new, and is not rooted in problems with machines, but instead with humans.
Ethical Resolve, Provides blog posts, talks, and resources for businesses concerned with implementing responsible AI and Ethics in Design.
Nature Machine Intelligence, The Global Landscape of AI Ethics Guidelines - Maps and analyzes the current corpus of AI ethics guidelines.
Partnership on AI, Report on Algorithmic Risk Assessment Tools in the U.S. Criminal Justice System - This report documents the serious shortcomings of risk assessment tools in the U.S. criminal justice system, most particularly in the context of pretrial definitions.
Access Now, Human Rights in the Age of Artificial Intelligence - Provides a jumping off point for further conversation and research in this developing space.
AI Now Institute, Algorithmic Accountability Policy Toolkit - This toolkit is intended to provide legal and policy advocates with a basic understanding of government use of algorithms including, a breakdown of key concepts and questions that may come up when engaging with this issue, an overview of existing research, and summaries of algorithmic systems currently used in government.
Data & Society, Algorithmic Accountability: A Primer - The primer explores the trade-offs debates about algorithms and accountability across several key ethical dimensions, including fairness and bias; opacity and transparency; and lack of standards for auditing.
Data & Society, Governing Artificial Intelligence: Upholding Human Rights & Dignity - This report shows how human rights can serve as a “North Star” to guide the development and governance of artificial intelligence.
Karen Hao, This is How AI Bias Really Happens—and Why It's So Hard to Fix - Bias can creep in at many stages of the deep-learning process, and the standard practices in computer science aren’t designed to detect it.
UNICEF & Human Rights Center, Executive Summary: Artificial Intelligence and Children's Rights - This memo outlines a series of case studies to illustrate the various ways that artificial intelligence-based technologies are beginning to positively and negatively impact children’s human rights.
FAT ML, Principles for Accountable Algorithms and a Social Impact Statement for Algorithms - The goal of this document is to help developers and product managers design and implement algorithmic systems in publicly accountable ways.
Salesforce, Ethical Frameworks, Tool Kits, Principles, and Certifications—Oh My! - This post links to dozens of ethics resources from both the private and public sector.
ICO, Draft Guidance for Consultation, Guidance on the AI Auditing Framework - advice on how to understand data protection law in relation to artificial intelligence (AI) and recommendations for organisational and technical measures to mitigate the risks AI poses to individuals.
European Commission, White Paper On Artificial Intelligence - A European Approach to Excellence and Trust - A European approach to artificial intelligence building off the European strategy for AI presented in April 2018.
European Union Agency for Fundamental Rights, Data Quality and Artificial Intelligence—Mitigating Bias and Error to Protect Fundamental Rights
Norwegian Data Protection Authority, Data Protection by Design and by Default
Berkeley Technology Law Journal, Rethinking Explainable Machines: The GDPR's 'Right to Explanation' Debate and the Rise of Algorithmic Audits in Enterprise
World Economic Forum, Top 9 Ethical Issues In Artificial Intelligence
Office of the Victorian Information Commissioner, Artificial Intelligence and Privacy
European Agency for Fundamental Rights, #Big Data: Discrimination in Data-Supported Decision Making
United Nations University, Centre for Policy Research, The Ethical Anatomy of AI
Australian Institute of Company Directors,Preparing Directors For Artificial Intelligence Whirlwind
Université de Montréal, Montréal Declaration of Responsible AI
House of Lords, Select Committee on Artificial Intelligence, AI in the UK: ready, willing, and able?
European Group on Ethics in Science and New Technologies, Statement on Artificial Intelligence, Robotics and 'Autonomous' Systems
European Commission for the Efficiency of Justice (CEPEJ), European ethical Charter on the use of Artificial Intelligence in judicial systems and their environment
European Commission High-Level Expert Group on Artificial Intelligence (AI HLEG), Draft Ethics guidelines for trustworthy AI
Automating Society, Taking Stock of Automated Decision-Making in the EU
UK Information Commissioner's Office,Automated Decision Making: the role of meaningful human reviews- This framework describes how humans can have ‘meaningful’ involvement in AI decision making.
Dr. Thilo Hagendorff, The Ethics of AI Ethics- An evaluation of the AI guidelines that have been presented. An overview of the field of AI ethics.
Government of Canada, Responsible use of artificial intelligence (AI) - Guiding principles to ensure the use of ethical AI by the government.
The White House, The Administration’s Report on the Future of Artificial Intelligence - A 2016 report focusing on the opportunities, considerations, and challenges of Artificial Intelligence.
Office of the Privacy Commissioner of Canada, Consultation on the OPC's Proposals for Ensuring Appropriate Regulation of Artificial Intelligence - proposals for regulating artificial intelligence.
II. Resources for AI Ethical Review Process
Future of Privacy Forum, Conference Proceedings - Beyond IRBs Designing Ethical Review Processes for Big Data Research - 2015 conference proceedings aimed at identifying processes and commonly accepted ethical principles for data research in academia, government and industry.
Northeastern University Ethics Institute, Building Data and AI Ethics Committees - Describes components of a committee-based approach to data and AI ethics, while identifying questions for an organization to consider when developing ethics and oversight committees.
Council of Europe, The Council of Europe Established an Ad Hoc Committee on Artificial Intelligence - CAHAI - Committee to examine the feasibility and potential elements of a legal framework for the development of artificial intelligence.
Berkman Klein Center, Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-Based Approaches to Principles for AI- Provides a comparison between thirty-six prominent AI principles documents side-by-side.
BNH.AI, Sample AI Incident Response Checklist - A checklist for 7 Phases of AI incident response including: preparation; identification; containment; eradication; and recovery. Also provides additional compliance resources.
Dallas Card & Noah A. Smith, On Consequentialism and Fairness - A consequentialist critique of common definitions of fairness within machine learning, as well as a machine learning perceptive on consequentialism; concluding with a broader discussion of how issues of learning and randomization have important implications for the ethics of automated decision making systems.