Autonomous CPS
Validation & Verification

How are we going to trust the autonomous vehicles?

In the world of Autonomous Vehicles, one of the emerging challenges is the merging of the decision taking with the driving mechanics. In this setting, the important validation issue becomes the scenario testing. Various studies show that the "real world" testing is extremely time consuming, ineffective, converges too slowly, and has no framework for certification. In addition to real-life testing, there are several autonomous vehicle testing methods such as VEhicle Hardware-in-Loop (VEHiL), Vehicle-in-the-loop (ViL) and driving simulators. All of these require a standard method for the identification of test scenarios and the reduction of sample space for possible scenarios is an urgent requirement for proving the systems to be safe in a time and cost efficient manner. Our AV testing research focuses on the formulation of the verification challenge for AVs and present a framework to address this challenge.

We are one of the founding partners of the Autonomous Vehicles Verification Consortium (AVVC). Together with our partners. we have been working on scenario-based verification and validation of autonomous vehicles (AVs). In particular, we have built PolyVerif, an open-source environment for the verification and validation (V&V) of AVs. We have been connecting three open-source projects together into a V&V toolkit: (i) Autoware, one of leading open-source software systems for autonomous driving; (ii) the SVL simulator developed by LG Silicon Valley Labs and CARLA, an open-source simulator, and (iii) the Scenic probabilistic programming system developed by Dr. Daniel Fremont (USSC) and Sanjit Seshia (UCB) (iv) SUMO open source, microscopic and continuous multi-modal traffic simulation package. We have recently developed an interface between Scenic and the SUMO traffic simulator (https://github.com/AkbasLab/scenic-sumo).  

Below, you can see my invited talk at FIAP, Sao Paulo, Brazil and a News 13 clip about our research when they visited my research group at my previous institution:

AV Proving Grounds

Autonomous Vehicle (AV) Technology has the potential to have a significant impact in various fields such as logistics, farming and transportation. NHTSA standards Level 1 and level 2 capabilities are already available. However, to reach the full potential of the technology, Level 3 and Level 4 autonomy must be realized. The verification and testing of AVs is critical in order to achieve full autonomy. Considering the active work ongoing with proving grounds for testing, we thought building a database of proving grounds would be useful to the broader Autonomous Vehicle community.  The database can be accessed with read privileges HERE.

AV Accidents

The existing AV accident data can be an important resource for AV analysis. We present our work of a deep dive analysis of all reported AV related accidents to date in an illustrative database (link).

Related Publications

• J. M. Thompson, Q. Goss and M. I. Akbas. "A Strategy for Boundary Adherence and Exploration in Black-Box Testing of Autonomous Vehicles ." Accepted to the IEEE International Conference on Mobility: Operations, Services, and Technologies (IEEE MOST), May, 2023.

• Q. Goss and M. I. Akbas. "Integration of Formal Specification and Traffic Simulation for Scenario-Based Validation ." Accepted to the IEEE International Conference on Mobility: Operations, Services, and Technologies (IEEE MOST), May, 2023.

• J. M. Thompson, Q. Goss and M. I. Akbas. "Boundary Adherence and Exploration in High Dimensions for Validation of Black-Box Systems." Accepted to the IEEE World Forum on Internet of Things (IEEE WFIoT2022), November, 2022.

• Q. Goss and M. I. Akbas. "Eagle Strategy with Local Search for Scenario Based Validation of Autonomous Vehicles." In the IEEE International Conference on Connected Vehicles and Expo (ICCVE), March, 2022.

• Q. Goss, Y. AlRashidi, and M. I. Akbas. "Generation of Modular and Measurable Validation Scenarios for Autonomous Vehicles Using Accident Data". In the IEEE Intelligent Vehicles Symposium (IV), pp. 251-257, July, 2021.

• A. J. Alnaser, A. Sargolzaei and M. I. Akbas. "Autonomous Vehicles Scenario Testing Framework and Model of Computation: On Generation and  Coverage.'' In IEEE Access, April, pp. 1-12, doi: 10.1109/ACCESS.2021.3074062, 2021.

• M. Aydin and M. I. Akbas. "Identification of Test Scenarios for Autonomous Vehicles Using Fatal Accident Data." In the Society of Automotive Engineers (SAE) International Journal of Connected and Automated Vehicles (JCAV), 4(1):2021. 

• M. I. Akbas. “Testing and Validation Framework for Autonomous Aerial Vehicles.” Accepted to the Journal of Aviation/Aerospace Education & Research (JAAER), 30(1), 2021. 

• R. Razdan, M. I. Akbas, Jim Cherian, Niels de Boer, B. Schmidt, R. Sell, C. Tino. "Unsettled Topics Concerning Autonomous Public Transportation Systems." No. EPR2020020. SAE EDGE™ Research Report, October, 2020.

• C. Medrano-Berumen and M. I. Akbas. “Validation of Decision Making in Artificial Intelligence Based Autonomous Vehicles.” Accepted to the Journal of Information and Telecommunication (JIT), Taylor & Francis, September, 2020.

• R. Razdan, W. Mahoney, E. Haritan, A. Kalia, J. Smith, T. Zarola, M. I. Akbas, Joachim Taiber, E. Straub, R. Sell. "Unsettled Topics Concerning Automated Driving Systems and the Transportation Ecosystem." No. EPR2020004. SAE EDGE™ Research Report, March, 2020.

• C. Medrano-Berumen and M. I. Akbas. “Scenario Generation for Validating Artificial Intelligence Based Autonomous Vehicles.” In the ACIIDS Special Session on Privacy, Security and Trust in Artificial Intelligence (PSTrustAI), March, Springer Nature, 2020. 

• C. Stark, C. Medrano-Berumen and M. I. Akbas. "Generation of Autonomous Vehicle Validation Scenarios Using Crash Data." In the IEEE SoutheastCon, March, 2020.

• C. Medrano-Berumen, M. Malayjerdi, M. I. Akbas, R. Sell and R. Razdan. "Development of a Validation Regime for an Autonomous Campus Shuttle." In the IEEE SoutheastCon, March, 2020.

• A. J. Alnaser, M. I. Akbas, A. Sargolzaei and R. Razdan. "Autonomous Vehicles Scenario Testing Framework and Model of Computation''. SAE (Society of Automotive Engineers) International Journal of Connected and Automated Vehicles, December, 2019.

• R. Razdan, W. Mahoney, E. Haritan, A. Kalia, J. Smith, T. Zarola, M. I. Akbas, Joachim Taiber, E. Straub, R. Sell. "Unsettled Topics Concerning Automated Driving Systems and the Transportation Ecosystem." No. EPR2019005. SAE EDGE™ Research Report, November, 2019.

• R. Razdan, M. I. Akbas, A. Sargolzaei, A. J. Alnaser, S. Sahawneh, S. Alsweiss, J. Vargas. "Unsettled Technology Areas in Autonomous Vehicle Test and Verification". SAE (Society of Automotive Engineers) EDGE™ Research Report EPR2019001, 2019.

• R. Razdan, J. Boxold, J. W. Taylor, R. Watts, G. Ralston, D. Neef, P. Mayor (M. I. Akbas and A. Sargolzaei as contributors). “Unsettled Topics Concerning Automated Driving Systems and the Transportation Ecosystem”. Society of Automotive Engineers (SAE) EDGE™ Research Report EPR2019005, 2019.

• C. Medrano-Berumen and M. I. Akbas. "Abstract Simulation Scenario Generation for Autonomous Vehicle Verification". In Proceedings of the IEEE SoutheastCon, April, 2019.

• S. Sahawneh, A. J. Alnaser, M. I. Akbas, A. Sargolzaei and R. Razdan. "Requirements for the Next-Generation Autonomous Vehicle Ecosystem ". In Proceedings of the IEEE SoutheastCon, April, 2019.

• J. Rentrope, M. Midence, M. I. Akbas and R. Razdan. “Verification of Autonomous Vehicles Using Simulations with String Based Scenarios”. In the Institute of Industrial & Systems Engineers (IISE) Annual Conference and Expo, May, 2018.