Full description of the incident
A plain-language description of the incident in one paragraph or less.
Up to one paragraph describing all known or suspected inputs which lead to the incident. This includes user inputs, like “turn left” and non-user inputs, like weather conditions, number, color, and size of nearby vehicles, time of day, etc.
Short description of the incident
A one-sentence description of the incident.
Timeline
When the AI incident began and when it ended
Description of AI system involved
A brief description of the AI system(s) involved in the incident, including the system’s intended function, the context in which it was deployed, and any available details about the algorithms, hardware, and training data involved in the system.
System developer
The entity that created the AI system.
Sector of deployment
The primary economic sector in which the AI system(s) involved in the incident were operating.
Information and communication
Arts, entertainment and recreation
Transportation and storage
Public administration and defense
Administrative and support service activities
Human health and social work activities
Education
Professional, scientific and technical activities
Financial and insurance activities
Wholesale and retail trade
Relevant AI functions
Indicates whether the AI system(s) were intended to perform any of the following high-level functions:
"Perception," i.e. sensing and understanding the environment;
"Cognition," i.e. making decisions;
"Action," i.e. carrying out decisions through physical or digital means.
Unclear
Location
The location or locations where the incident played out
AI is certain about an incorrect result
AI is repeatedly uncertain in a situation which requires certainty
Uncertainty bounds are too wide for given task
The problem at hand cannot be solved well by an AI
Such as complex mathematics or long-term market prediction
The AI’s purpose is malevolent, or has a gross lack of consideration for potential consequences. It has failed humanity even if (or especially if) it behaves as intended.
The AI is perfectly well and good, but was used in a context it was not designed for
Fault of employees / managers / was the AI mis-promoted?
The AI did not have a failsafe
Or one which could be deployed sufficiently quickly in case things went wrong
The AI’s objectives are changed, leading to Catastrophic Forgetting
A likely use case was not considered before deployment
A human shares responsibility for the incident
The algorithm uses an insufficient stand-in for its target data
Weakness in Algorithm:
The algorithm can not incorporate training data to the fullest extent, leading to shortcuts in learning and a weak algorithm overall. Notably, rewards based learning can encourage this behavior when specification for a task is underspecified. This category, in isolation of other ‘weak’ causes, is for non-ML AI failures. ML AIs which fall under this category often have insufficient specification or an inadequate approach for their given task, and will fall under one or more other ‘weak’ causes.
Weakness in Training:
The developer does not include sufficient training data to train the AI learning model.
Known or suspected weaknesses in the training data, ex:
Bias
All data x also shared y quality, when z quality also a possibility
Incorrect data that was used in training
Result of adversarial attack or likely negligence?
Uncovered Situations
Data x was not covered in its entirety
Possibly due to too small a set of training data in the first place
Insufficient time dedicated to training
Wrong learning model used for training
Weakness in Testing:
The test suite the AI was tested on did not cover a combinatorially significant proportion of potential RW cases, or fails to test a probable use case. We recommend that all ML AI be tested using combinatorial testing, which has been proven to catch more bugs than random testing at a fraction of the number of test cases.
Known or suspected weaknesses in the testing data, ex:
Biases Present
Incorrect tests used in testing
Uncovered Situations
Insufficiently large testing suite in the first place
Insufficient time dedicated to testing
Weakness in Defense:
An adversarial attack is able to trick the AI by exploiting weakness in its model. In the case of Neural network based learning models, it attacks the network’s way of understanding input. This must be a) a deliberate attack and b) cause the AI to act in an unintended way.
Known or suspected means of adversarial attacks, ex:
Noise Attack
Imperceptible noise is added to an input in an attempt to trick the AI into misclassifying it.
“Physical” Attack
One simple way in which the query input is changed from x to x′ is by simply adding something physically (eg. bright color) to disturb the model.
Another example comes from researchers at Google who added stickers to the input image to change the classification of images.
Out of Distribution (OOD) Attack
The traditional assumption in machine learning is that all train and test examples are drawn independently from the same distribution. In an OOD attack, this assumption is exploited by providing images of a different distribution from the training dataset to the model.
Other Attacks