RQ1: Can LeGEND improve diversity of
scenarios, compared to baseline approaches?
Embedded Files
Experiment Design
To address RQ1, we perform a comparative analysis with two baseline approaches: Random and AV-Fuzzer. We run LeGEND, Random, and AV-Fuzzer for the same number of simulations and record the critical scenarios that contain collisions between ego and NPC vehicles. These critical scenarios are classified into different types based on their logical patterns, i.e., the actions of the vehicles prior to each collision. We then measure their performance in terms of the following metrics:
• The number of distinct types of critical scenarios, denoted as #Types;
• The exposure rate of distinct types over the identified critical scenarios, denoted as #TypeExposRate;
• The number of simulations required to detect the first type of critical scenarios, represented by #SimForFirstType;
• The number of simulations required to detect all types of critical scenarios, denoted as #SimForAllTypes;
• The time cost for one scenario, denoted as #TimeForOneScenario.
Comparison Results
The growth in the number of critical scenario types discovered by the three approaches over simulations is shown in the top left figure. The distribution of critical scenarios generated by LeGEND across each type is shown in the bottom left figure. We can observe that LeGEND outperforms the other methods since it manages to detect the highest number of distinct types of critical scenarios. Moreover, LeGEND continuously exposes new types of critical scenarios throughout the simulations. In comparison, AV-Fuzzer exhibits the fastest convergence but detects only five distinct types of critical scenarios. Random exhibits a similar convergence speed to AV-Fuzzer but identifies the fewest types of critical scenarios. Notably, all types of critical scenarios detected by both AV-Fuzzer and Random are exposed within the first 700 simulations.
The growth in the number of critical scenario types discovered by the three approaches over simulations is shown in the top left figure. The distribution of critical scenarios generated by LeGEND across each type is shown in the bottom left figure. We can observe that LeGEND outperforms the other methods since it manages to detect the highest number of distinct types of critical scenarios. Moreover, LeGEND continuously exposes new types of critical scenarios throughout the simulations. In comparison, AV-Fuzzer exhibits the fastest convergence but detects only five distinct types of critical scenarios. Random exhibits a similar convergence speed to AV-Fuzzer but identifies the fewest types of critical scenarios. Notably, all types of critical scenarios detected by both AV-Fuzzer and Random are exposed within the first 700 simulations.
The comparative analysis between LeGEND and Random is presented in the left table. Over the period of 1400 simulations, Random generates 64 critical scenarios that reveal potential system faults, with the first critical scenario identified during the 34-𝑡ℎ simulation. It can identify 2 types of critical scenarios during the scenario generation process. All types of the critical scenarios are observed within the first 642 simulations. Compared to Random, LeGEND generates a larger number of critical scenarios and uncovers a greater diversity of them. The type-exposure frequency #TypeExposRate of LeGEND is higher (avg 11.5%), which indicates its ability to generate diverse critical scenarios effectively. Additionally, the two types of critical scenarios identified by Random are also detected by LeGEND in the 118-𝑡ℎ and 497-𝑡ℎ simulation respectively.
The comparative analysis between LeGEND and Random is presented in the left table. Over the period of 1400 simulations, Random generates 64 critical scenarios that reveal potential system faults, with the first critical scenario identified during the 34-𝑡ℎ simulation. It can identify 2 types of critical scenarios during the scenario generation process. All types of the critical scenarios are observed within the first 642 simulations. Compared to Random, LeGEND generates a larger number of critical scenarios and uncovers a greater diversity of them. The type-exposure frequency #TypeExposRate of LeGEND is higher (avg 11.5%), which indicates its ability to generate diverse critical scenarios effectively. Additionally, the two types of critical scenarios identified by Random are also detected by LeGEND in the 118-𝑡ℎ and 497-𝑡ℎ simulation respectively.
The comparison between LeGEND and AV-Fuzzer is detailed in the left table. During the 12-hour execution, AV-Fuzzer generates 297 critical scenarios with the first collision occurring in the 14-𝑡ℎ simulation. The high number of identified critical scenarios by AV-Fuzzer can be attributed to the local search mechanism employed by this technique. However, despite the large number of critical scenarios, AV-Fuzzer is limited to exposing only 4 distinct types, resulting in a type exposure frequency of 1.3%. The four types of critical scenarios are all exposed by LeGEND in the 118-th, 411-𝑠𝑡, 497-𝑡ℎ and 822-𝑛𝑑 simulation. Compared to AV-Fuzzer, all four types of critical scenarios identified by AV-Fuzzer are also detected by LeGEND. Moreover, LeGEND exhibits a greater diversity of distinct critical scenarios during the critical scenario generation process. The type exposure rate for LeGEND is 11.5%, significantly higher than the 1.6% observed with AV-Fuzzer. Furthermore, LeGEND detects the first critical scenario more quickly and has a lower time cost per scenario, compared to AV-Fuzzer.
The comparison between LeGEND and AV-Fuzzer is detailed in the left table. During the 12-hour execution, AV-Fuzzer generates 297 critical scenarios with the first collision occurring in the 14-𝑡ℎ simulation. The high number of identified critical scenarios by AV-Fuzzer can be attributed to the local search mechanism employed by this technique. However, despite the large number of critical scenarios, AV-Fuzzer is limited to exposing only 4 distinct types, resulting in a type exposure frequency of 1.3%. The four types of critical scenarios are all exposed by LeGEND in the 118-th, 411-𝑠𝑡, 497-𝑡ℎ and 822-𝑛𝑑 simulation. Compared to AV-Fuzzer, all four types of critical scenarios identified by AV-Fuzzer are also detected by LeGEND. Moreover, LeGEND exhibits a greater diversity of distinct critical scenarios during the critical scenario generation process. The type exposure rate for LeGEND is 11.5%, significantly higher than the 1.6% observed with AV-Fuzzer. Furthermore, LeGEND detects the first critical scenario more quickly and has a lower time cost per scenario, compared to AV-Fuzzer.
All Types of Critical Scenarios Exposed by LeGEND
All Types of Critical Scenarios Exposed by LeGEND
Example of the 1st type
Example of the 1st type
Description: the ego vehicle is driving behind two NPC vehicles on a straight road. Suddenly, vehicle B starts to decelerate, causing vehicle A to brake as well. However, the ego vehicle fails to decelerate promptly, leading to a collision.
Example of the 2nd type
Example of the 2nd type
Description: the ego vehicle is driving forward while an NPC vehicle A drives on the adjacent lane. Then, vehicle A starts to change lane into the ego and decelerates before the change is completed. However, the ego vehicle fails to react in time and collides with the vehicle A.
Example of the 3rd type
Example of the 3rd type
Description: the ego vehicle is driving between two NPC vehicles on a straight road. Suddenly, vehicle B starts to decelerate, causing the ego vehicle to change lane. However, the ego vehicle brakes during the change and collides with the vehicle A from behind.
Example of the 4th type
Example of the 4th type
Description: the ego vehicle is traveling on a straight road
with two NPC vehicles A and B driving on the adjacent two lanes. Suddenly, vehicle B attempts to change lanes into A, and the
ego vehicle detects it by slightly braking. During B’s lane change, vehicle A decides to move into the lane that the ego drives. However, the ego vehicle continues driving forwards and fails to respond timely to vehicle A’s lane change, leading to a collision.
Example of the 5th type
Example of the 5th type
Description: the ego vehicle is driving forward while an NPC vehicle A drives on the rightmost lane. Suddenly, vehicle A changes into the middle lane and then changes into the ego's lane. However, the ego vehicle fails to decelerate in time and collides with the vehicle A.
Example of the 6th type
Example of the 6th type
Description: the ego vehicle is driving behind of two NPC vehicles on a straight road. The ego vehicle decides to overtake the vehicle A due to its low speed. However, when the ego cuts into the vehicle A's lane, it collides with the vehicle A.
Example of the 7th type
Example of the 7th type
Description: the ego vehicle is driving on a curve road, with an NPC vehicle A driving on the left side. Then, vehicle A starts to cut into the ego's lane. However, the ego vehicle fails to decelerate in time while maintaining the steering, leading to a collision.
Example of the 8th type
Example of the 8th type
Description: the ego is driving on a curve road, with an NPC vehicle A driving on the left side and an NPC vehicle B driving on the right side. Vehicle B starts to cut into the ego's lane. After that, vehicle B cuts out with a deceleration. However, the ego vehicle fails to react in time and collides with vehicle B.
Example of the 9th type
Example of the 9th type
Description: the ego is driving on a curve road, with an NPC vehicle A driving on the right side. Suddenly, vehicle A starts to cut into the ego's lane, and after the cut-in, it maintains a lower speed. However, the ego vehicle fails to keep the safe distance with A and collides with its rear side.
Example of the 10th type
Example of the 10th type
Description: the ego is driving on a curve road, with an NPC vehicle A driving behind it. Then, vehicle A overtakes the ego vehicle and drives in front of the ego vehicle after the overtaking. However, the ego vehicle fails to react promptly and collides with vehicle A.
Example of the 11th type
Example of the 11th type
Description: the ego is driving on a curve road, with an NPC vehicle A driving in the innermost lane. Then, vehicle A changes to the middle lane and then cuts in front of the ego vehicle. However, the ego vehicle fails to decelerates while maintaining the steering, leading to a collision.
Google Sites
Report abuse