Background Information:

• Full description of the incident

  • The March 2018 accident was the first recorded death by a fully autonomous vehicle. On-board video footage showed the victim, 49-year-old Elaine Herzberg, pushing her bike at night across a road in Tempe, Arizona, moments before she was struck by the AI-powered SUV at 39 MPH. Uber’s AI repeatedly misclassified Herzberg as a car, a bike, and an unknown object before determining too late that a crash was unavoidable. The SUV’s Driver, Rafaela Vasquez, was using Uber’s “autonomous mode”, and was visibly not paying attention to the road.

  • There were no human inputs which affected the crash, though human inputs from the driver may have prevented it. Other inputs which may have affected the AI was that it was a new moon, Herzberg’s bike had shopping bags hanging from the handlebars, Herzberg was jaywalking, and that the SUV’s auto-braking feature was disabled.

• Short description of the incident

  • Uber’s self-driving car was involved in a fatal accident after not being able to recognize a pedestrian crossing the road.

• Timeline

  • The incident occurred on Sunday March 18, 2018 at 9:58 pm

• Location

  • On a highway in Tempe, Arizona

• Description of AI system involved

  • Uber’s “autonomous mode,” a ML algorithm meant to allow for autonomous driving, i.e. it will drive the car instead of the driver.

• System developer

  • Uber

• Sector of deployment

  • Transportation and storage

• Relevant AI functions

  • "Perception," i.e. sensing and understanding the environment;

  • "Cognition," i.e. making decisions;

  • "Action," i.e. carrying out decisions through physical or digital means.

Causes

Causes for Incident:

• AI is repeatedly uncertain in a situation which requires certainty

  • The AI repeatedly misclassified Herzberg as a bike, car, and unknown object before colliding. For a autonomous AI to be safe, it must be certain of what is around it.

• AI did not have a failsafe which deployed in time

  • The AI's failsafe was to delay braking for a full second while returning control of the vehicle to the driver. However, this activated with less than a second before an unavoidable collision, making the failsafe insufficient. Furthermore, the SUV's internal failsafe, which would have braked to reduce the damage from a collision, was disabled since it conflicted with Uber's autonomous mode.

• A likely use case was not considered before deployment

  • Jaywalking pedestrians were not considered when developing the autonomous mode.

• A human has partial responsibility for the incident

  • The driver was watching "The Voice" at the time of the incident, and was distracted.

Sources of Weakness:

Weakness in Algorithm

• Objects can be classified in quick succession, preventing the AI from tracking location and direction of movement.

Weakness in Training - Biases Present:

• The training was biased towards vehicles moving in the same direction as the user’s car when in the same lane as the user’s car.

• Pedestrians were only present on crosswalks in the training data.

Weakness in Training - Uncovered Situations

• Uber’s AI was not trained on jaywalking pedestrians, despite this being a likely RW case.

Weakness in Testing - Uncovered Situations

• Uber’s AI was not tested on jaywalking pedestrians, despite this being a likely scenario.

Further Research

• https://www.ntsb.gov/investigations/accidentreports/reports/har1903.pdf

• https://www.ntsb.gov/news/events/Pages/2019-HWY18MH010-BMG.aspx

• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4430591/